I'm kind of at a crossroads in regards to this as well, ficbot.
Right now I'm leaning towards using Stanza for PDFs, and iBooks for ePubs. Which means Calibre has both PDFs and ePubs in its library, iBooks has the ePubs duplicated in its iTunes Media Library, and Stanza relying on Calibre's library to store the on-computer PDF library.
I don't notice too much of a slowdown with Calibre-but I don't have anywhere near your database...I'm completely re-doing my Calibre library, and probably will end up with 65-70 ePubs and 10-15 PDFs.
The other option is to have a third, program neutral file library stored somewhere else, with the different formats saved out. I've just been so used to using Calibre's one library (shared on Dropbox with 2 computers), that having iBooks/iTunes as another library has made things a bit odd.
I'm curious as to how other people are storing/cataloging their eBook collections outside their devices. In some ways, pure Kindle/Nook users have it simple-let Amazon and B&N deal with it

.