Performance with large Libraries?

itimpi · 12-14-2008, 06:49 AM

I am finding that as my library is getting larger, the performance of Calibre seems to be dropping of markedly when adding new books. As I want to add several thousand more this is worrying as it might mean Calibre becomes unusable for a Library of this size.

In case it matters, I am running under Windows XP, with the Calibre library on a network drive, and currently around 1000 books stored in it.

Questions I have are:
- Is this a known issue?
- Is it specific to Windows?
- Is there anything I can do to improve performance?
- Is there anything I can do to diagnose the bottleneck?

Mitchll · 12-14-2008, 09:01 AM

My library totals over distinct 4000 books, with multiple formats for each. Calibre's performance does not appear to degrade except that sorts and startup are slow so as to account for the volume of the search.

Mitch

kovidgoyal · 12-14-2008, 10:38 AM

Hmm IIRC adding doesn't involve any operations that scale non-linearly with database size. If you have the source, adding a few

Code:

from time import time
print time.time()

in the _add_books function in gui2/main.py should show you where the slowdown is occurring (you will have to run the GUI using

Code:

calibre-debug -c "from calibre.gui2.main import main; main()"

to see the output

itimpi · 12-14-2008, 02:46 PM

OK - that gives me something to work with.

I thought that with the metadata in a database it should scale reasonably well as the metadata.db file is not that large (only a few megabytes), but as I was having problems I though I would check. I was also concerned that I might be encountering an issue that was Windows specific.

Anyway I will report back when I have more information. It may well be something local to my system and not related Calibre at all.

12-14-2008, 06:49 AM	#1
itimpi Wizard Posts: 4,552 Karma: 950151 Join Date: Nov 2008 Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)	Performance with large Libraries? I am finding that as my library is getting larger, the performance of Calibre seems to be dropping of markedly when adding new books. As I want to add several thousand more this is worrying as it might mean Calibre becomes unusable for a Library of this size. In case it matters, I am running under Windows XP, with the Calibre library on a network drive, and currently around 1000 books stored in it. Questions I have are: - Is this a known issue? - Is it specific to Windows? - Is there anything I can do to improve performance? - Is there anything I can do to diagnose the bottleneck?

12-14-2008, 10:38 AM	#3
kovidgoyal creator of calibre Posts: 43,850 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Hmm IIRC adding doesn't involve any operations that scale non-linearly with database size. If you have the source, adding a few Code: from time import time print time.time() in the _add_books function in gui2/main.py should show you where the slowdown is occurring (you will have to run the GUI using Code: calibre-debug -c "from calibre.gui2.main import main; main()" to see the output

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
For those with large (500+) libraries...	ficbot	General Discussions	60	12-13-2011 01:51 PM
Scalability and performance	blither	Calibre	1	08-10-2010 08:31 PM
DR800 Performance	jjansen	iRex	18	08-03-2010 09:10 PM
DX performance on large (>100MB) pdf files?	Blackguard	Amazon Kindle	6	06-16-2009 06:57 PM
Library size and performance	MrFinch	Calibre	4	01-01-2009 04:11 AM

12-14-2008, 09:01 AM	#2
Mitchll Cave Dweller Posts: 220 Karma: 1986 Join Date: Oct 2006 Device: Ipad Sony reader prs-505 prs-900	My library totals over distinct 4000 books, with multiple formats for each. Calibre's performance does not appear to degrade except that sorts and startup are slow so as to account for the volume of the search. Mitch

12-14-2008, 02:46 PM	#4
itimpi Wizard Posts: 4,552 Karma: 950151 Join Date: Nov 2008 Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)	OK - that gives me something to work with. I thought that with the metadata in a database it should scale reasonably well as the metadata.db file is not that large (only a few megabytes), but as I was having problems I though I would check. I was also concerned that I might be encountering an issue that was Windows specific. Anyway I will report back when I have more information. It may well be something local to my system and not related Calibre at all.

Advert