I have what I consider a "big" library (~500 titles, its an addiction) on my K3, but not a "massive" library of several thousand. My previous device, a Sony Clie UX50, generally had 50-80 titles on it for most of the 8 years or so that it served as my primary ebook reader.
I generally keep a stack of titles in a "waiting to be read" pile. With the old UX50, I didn't load more than 100 on the device, holding the rest on my PC. When I got my K3 I simply moved the files from the UX50 and added the whole waiting folder from the PC (at my age I'll be lucky to live long enough to read the 500 or so already on my K3, much less the ones I'll undoubtably add as time goes by...). When I've finished reading a book I generally delete it. I don't keep many titles on my device that I've read, typically only ones that are part of a series that I may need to refer back to when reading the subsequent volumes. I keep backups on my PC for re-reading if the desire should occur. I have a rather large number of titles (book length and short works) archived on my PC
With the K3, I use the new "Collections" to provide some sanity when dealing with large libraries. I have collections arranged by genre, by type/length, and have a "short list" collection for queuing up the next few to be read. By not using a folder based collection system, the K3 allows the flexibility of placing the same book into multiple collections. I have some books in "SF & F" (Science Fiction & Fantasy), "Short Story", and "Short List" at the same time.