Couple of quick questions.
What OS are you running calibre under? If Windows 64 bit are you using the 64 bit version of calibre?
Are you converting the books on the fly or are they all in mobi/KF8/KFX format in your library?
If KF8, if you open one of them in calibre's ebook-editor, does the error check show any errors?
Edit: You might also want to take a look at HarryT's
Tips for loading a large library onto a KO2 thread. A bit old but most of it still applies especially the indexing wait times.