Scalability and performance

blither · 08-10-2010, 07:42 PM

I've been using Calibre for over a year with my Sony PRS-505. Actually, I read about Calibre and that's what helped me pick the Sony...

Anyway, my ebook collection is pretty large; over 20,000 books. I have most of the Gutenberg titles, lots of PDFs and RTF/DOCs from work, and LIT/ePUBs that I have purchased. I have been converting as much as possible into ePUB, which has gone pretty well. I've got Calibre running on Windows and on Linux (catalogs synched with DropBox).

My question is about performance and scalability tweaks in Calibre.

Searching and filtering is pretty good - if your tags are up to date. Even with a lot of tags on a lot of books, Calibre does a good job of sorting, which is important with a big library. But when I want to select a bunch of books to edit their tags, the process takes a LONG time. I'm still watching my system churn as it tries to add one new tag to 300 books that I just selected. It's been chugging along for about 10 minutes and will probably go for another 10 minutes.

I'm not complaining, because Calibre *works*! It may take a while, but it gets through the job successfully - eventually. My concern is about large jobs in general. I have done things like export the entire library and try to import it. The export went OK, but I can't import more than about 8000 books without Calibre crashing. It gets to one title or another and hangs for a while before eventually keeling over.

My library might not be typical, so any suggestions/recommendations for how to optimize performance with a large library would be appreciated...

kovidgoyal · 08-10-2010, 08:31 PM

Interesting, adding tags to multiple books is slow. That is probably something that can be optimized. Opena ticket for it and I'll look into it.

As for import, I wouldn't recommend importing more than about a 1000 books a shot. That is because importing is a very messy process, calibre has to read metadata from book files and for some formats (particularly PDF) calibre relies on third party libraries to do the heavy lifting, not all of which are as stable/leak proof as I would like.

08-10-2010, 07:42 PM	#1
blither Junior Member Posts: 1 Karma: 10 Join Date: Aug 2010 Device: PRS-505	Scalability and performance I've been using Calibre for over a year with my Sony PRS-505. Actually, I read about Calibre and that's what helped me pick the Sony... Anyway, my ebook collection is pretty large; over 20,000 books. I have most of the Gutenberg titles, lots of PDFs and RTF/DOCs from work, and LIT/ePUBs that I have purchased. I have been converting as much as possible into ePUB, which has gone pretty well. I've got Calibre running on Windows and on Linux (catalogs synched with DropBox). My question is about performance and scalability tweaks in Calibre. Searching and filtering is pretty good - if your tags are up to date. Even with a lot of tags on a lot of books, Calibre does a good job of sorting, which is important with a big library. But when I want to select a bunch of books to edit their tags, the process takes a LONG time. I'm still watching my system churn as it tries to add one new tag to 300 books that I just selected. It's been chugging along for about 10 minutes and will probably go for another 10 minutes. I'm not complaining, because Calibre works! It may take a while, but it gets through the job successfully - eventually. My concern is about large jobs in general. I have done things like export the entire library and try to import it. The export went OK, but I can't import more than about 8000 books without Calibre crashing. It gets to one title or another and hangs for a while before eventually keeling over. My library might not be typical, so any suggestions/recommendations for how to optimize performance with a large library would be appreciated...

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
DR800 PAGE TURN PERFORMANCE	musteman	iRex	24	09-03-2010 03:23 PM
DR800 Performance	jjansen	iRex	18	08-03-2010 09:10 PM
Classic Battery performance after softroot?	Sylver	Barnes & Noble NOOK	14	07-26-2010 10:51 PM
Performance in Bulk Metadata Changes	pfooti	Calibre	3	01-02-2010 07:59 PM
Calibre performance over the network	mrwindex	Calibre	11	07-30-2009 12:45 AM

08-10-2010, 08:31 PM	#2
kovidgoyal creator of calibre Posts: 43,843 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Interesting, adding tags to multiple books is slow. That is probably something that can be optimized. Opena ticket for it and I'll look into it. As for import, I wouldn't recommend importing more than about a 1000 books a shot. That is because importing is a very messy process, calibre has to read metadata from book files and for some formats (particularly PDF) calibre relies on third party libraries to do the heavy lifting, not all of which are as stable/leak proof as I would like.

Advert