Dealing with very large library

[Torpido]

I've just passed 150,000 ebooks in my calibre library. I moved the database to my local SSD drive, so it starts up fairly quickly (about 30-45 seconds), but I'm starting to have a problem with the actual ebook files. The problem is that there are over 65,000 directories in my calibre folder. It's getting impossible to manage that many subdirectories. It's taking longer and longer to batch add more ebooks. (I've managed to speed up the conversion by a factor of 50 by just making the 'temp' drive a ram disk (16GB), so I can convert them all to MOBI (I have a kindle fire), but the process of adding another 5000 books today took almost 2 hours. I've been thinking of moving the calibre folder to my NAS device, because it can handle millions of files without slowing down, but I did a test, and estimate it would take me close to 100 hours to move them all, don't want to waste 5 days if I don't see a good speed improvement.

[kovidgoyal]

I doubt filesystem access times are the limiting factor. When adding, the entire existing library has to be searched for duplicates, which is likely the biggest thing slowing down adds as library size grows.

[itimpi]

Originally Posted by Torpido

I've been thinking of moving the calibre folder to my NAS device, because it can handle millions of files without slowing down, but I did a test, and estimate it would take me close to 100 hours to move them all, don't want to waste 5 days if I don't see a good speed improvement.

Running Calibre off a NAS or any sort of networked drive is an unsupported configuration Many, including myself, have made it work but if things go wrong in that configuration you will not get much support.

I have in the past successfully had Calibre running with the database held locally and the main library files on the NAS. However in my experience it was significantly slower than holding all the files locally. I now run with the library held locally and make regular use of FreeFileSync to back it up to my NAS instead, and like you I use a RAM disk for temporary files.

[Torpido]

That's my fear. Since I have 50,000+ folders on the local hard drive, it's going to take forever to move them all.
But, I did come across what appears to be a great solution.
atpalca/docker-rdp-calibre.

I've got a DS3615 with 11x8TB drives. (about 75TB). Problem is my local storage HD is only 1TB, and my library is already over 300GB. It's not the fastest drive (I use SSD for system drives), but I'm almost out of space. Since I'm about to import 2015 (about 100GB) into the library, I wanted to find a fix before I do that. (I estimate about 75,000-100,00 ebooks).

So far, apart from a weird permissions problem I'm having on docker-rdp-calibre (I can't seem to point calibre to the folder I want to use as a library, it keeps showing me 'nobody', and 'nobody' can't browse to the volume that has the directory I created to store all the books.

The storage array has plenty of CPU power (think it's like a dual core 3Ghz, and 16GB of ram AND a 500GB SSD 'cache' drive, so I always get 100MB/sec transferring to and from it)