For the Kindle you would mostly buy contemporary non-fiction and fantasy from the Kindle Store. For the Sony you can buy from the Sony Store or from the many ebook stores that sell Adobe ePub and PDF ebooks. The Kindle store has the best non-fiction selection of any ebook store by a long way. However, some older non-fiction ebooks are only available as Adobe PDF ebooks which don't work on the Kindle.
For classics, these are available in MOBI format for the Kindle and in ePub for the Sony, and in any case Calibre (has its own forum here) can convert many DRM-free (unencrypted) ebook formats to MOBI or ePub.
All ebooks are loaded via USB on the Sonys and almost all ebooks except those directly from Amazon are loaded via USB on the Kindle also.
For either one you need a cover to protect the screen (the Sony comes with a sleeve).
The lowest cost EInk Reader in the US is currently the EZ Reader Pocket Pro deal from Books on Board, for example in
Black for $209 plus $10 for ebooks and 50% of the first 10 ebooks you buy. Other colors are available (look under Back to Physical Products), and they all come with a brown fake leather cover which I think is "good enough" looking and gives excellent protection. Astak has a similar bundle if you buy directly from them for $239 with money going to charity. The Pocket Pro reads the same Adobe ePub and PDF as the PRS-300 and is otherwise a comparable device with more expandability. You can buy ebooks for it from Books on Board and from the Sony ebook store and many other sites.