On a single ereader, why would you want more than 100 books? I have 5000 ebooks in my Calibre Library on the PC but but on my kindle I have four collections: 50 unread novels, 20 humor books, 20 non-fiction, and 8 reference.
These numbers fluctuate as I read through the books and I add to the ereader as I come across anything new that I want to read right away.
Unless you're kidnapped and held in a cave somewhere with your charger and an electrical outlet, it's unlikely that you need many more books than this on any portable device at any given time. Moreover, Calibre has a far superior library management capability on a PC than that found on any dedicated ereader.
By limiting the number of books on your device and ensuring that you have selected "manual management" in Preferences > Sending data to devices, I would hope that your load times were measured in a few seconds per upload.
Of course I'm no expert and only relating my experience with this excellent software when used with two kindle 3's, a Sony PRS 350, and a Hipstreet Reader. In all cases metadata transfers take seconds per book.