Calibre performance over the network

mrwindex · 06-10-2009, 12:19 AM

Hi,

I have read in the past about some users having difficulties with calibre and poor network performance. I am having this problem myself.
The last beta version is definitely not addressing any of these issues.
Calibre is having extremely low performance and most of the time over 60 seconds delay to tag a file with ISBN and cover data.

Is there a "fix" or workaround for these issues?

Using:
Calibre 0.6.0b2, before 0.5.14
Windows XP SP3
AVG Internet Security 8.0
hp xw8400 dual Xeon 3.2 Ghz 4Gb ram

EX487 with gigabit Ethernet
Library size: about 6000 pdf and djvu files

itimpi · 06-10-2009, 01:00 AM

Not that I know of.

I have reverted to having my Calibre files held locally, and then using RichCopy to keep my network copy synchronised with my local copy. The increase in Calibre perfomance is of the order of 100 times when running locally so it is well worth the overheads of running Richcopy periodically.

From what I have read elsewhere I suspect that the issue may well be Sqlite performance across networks, particularily running off Samba shares. It could well be worth carrying out s systematic timing exercise to prove if this is the issue.

tklaus · 06-10-2009, 01:12 AM

I've noticed the same thing, and I also have a gigabit network with a NAS accessed via Samba (NAS is Linux).

SQLLite sounds like a plausible culprit, is there any way to configure Calibre to store the database file on a different drive? I don't have enough local disk space for all my books, but the database file is small.

tklaus · 06-10-2009, 01:36 AM

Or if someone can point me to a location in the source where I can hack an alternate path to the metadata.db file, I'll try this experiment myself.

gwynevans · 06-10-2009, 04:18 AM

Quote:

Originally Posted by tklaus

Or if someone can point me to a location in the source where I can hack an alternate path to the metadata.db file, I'll try this experiment myself.

Probably the line "self.dbpath = os.path.join(library_path, 'metadata.db')" in "LibraryDatabase2.__init__" in "calibre.library.database2.py" is where you need to start. Try commenting that out & setting an absolute path to your local 'metadata.db' & see if that works/makes any difference.

To test, you'll either need to build calibre fully from source (best for debugging, etc) or failing that, get the source that matches your released version and use calibre-debug to replace the frozen module (in library.zip - take your own backup ) in the distribution.

tklaus · 06-10-2009, 05:13 PM

Thanks, I'll try that out and report back.

tklaus · 06-10-2009, 08:04 PM

One more question (I'm new to python), where does the output of the print statements go? Is there a log file somewhere, or is there a command-line flag to enable more verbose output? I'm using the binary distribution and I used calibre-debug to replace database2.py with my modified version.

Thanks!

p.s., I love Calibre, it's a great program, plus it gives me an excuse to learn Python, which has been on my todo list for a while....

kovidgoyal · 06-10-2009, 08:32 PM

If you're on windows run the GUI as

calibre-debug -g

from a command prompt

Then the output of the print statements will show up in the command prompt.

tklaus · 06-11-2009, 01:51 AM

With gwynevans pointer, I modified database2.py to specify a local (non-network) drive as the location of the metadata.db file as follows:

Code:

self.dbpath = os.path.join('c:\\calibre\\db', 'metadata.db')

The location of the library itself (with all the books) and the directory I was importing from remained on a NAS drive connected via a local gigabit network. The NAS drive is RAID 5 (striping with parity; 4 physical drives) where I get 7 MB/sec sustained transfer rates. The local disk is not RAID, and is actually slower than the NAS in many scenarios (4.5 MB/sec using the same benchmark).

For both tests, I started with an empty library (no files, no metadata.db file). I used Calibre 0.5.14 on Windows XP. For the test, I imported a directory tree containing 753 ebooks (mostly .lit).

Before the change, it took 54 minutes to import the library.
After the change, it took 10 minutes to import the library.

In both cases, the vast majority of the time was spent adding the books to the database (after scanning the files and reading the metadata), so I definitely suspect sqllite performance over SMB. I also noticed that the import time does not scale linearly with the number of books (size of the database) when the metadata.db file was on the NAS. The first 20% or so (according to the progress bar) took about the same amount of time in both cases, but the NAS case was really crawling by the end. Yesterday, it took over 4 hours to import 1600 books on the NAS (I'm not sure exactly how long it took because I went to bed), again starting with an empty database.

Kovid, are you open to adding an optional config parameter to specify an alternate location for metadata.db for NAS users like us?

Thanks,
Todd Klaus

kovidgoyal · 06-11-2009, 02:14 AM

Hmm open a ticket and I'll add a config option (you'll have to edit the calibre config files by hand to set it as I'm not going to implement a GUI for it).

tklaus · 06-11-2009, 03:33 AM

Done, thanks! Ticket #2575. None of the listed components seemed appropriate, so I left it as 'Graphical User Interface'

elfchief · 07-30-2009, 12:45 AM

Is there any ETA on having this implemented? It looks like there's a config file option for it now, but looking at the code, it seems to ignore it right now.

I, too, have discovered and tracked down the horrible, horrible performance of sqlite over smb, and it's making Calibre almost useless to me right now. I'd love to actually be able to, y'know, use it.

Willing to beg if needed!

06-10-2009, 12:19 AM	#1
mrwindex Junior Member Posts: 1 Karma: 10 Join Date: Jun 2009 Device: iLiad V2	Calibre performance over the network Hi, I have read in the past about some users having difficulties with calibre and poor network performance. I am having this problem myself. The last beta version is definitely not addressing any of these issues. Calibre is having extremely low performance and most of the time over 60 seconds delay to tag a file with ISBN and cover data. Is there a "fix" or workaround for these issues? Using: Calibre 0.6.0b2, before 0.5.14 Windows XP SP3 AVG Internet Security 8.0 hp xw8400 dual Xeon 3.2 Ghz 4Gb ram EX487 with gigabit Ethernet Library size: about 6000 pdf and djvu files

06-11-2009, 01:51 AM	#9
tklaus Connoisseur Posts: 71 Karma: 131 Join Date: Nov 2007 Location: San Jose, CA Device: iPad, Kindle DX	With gwynevans pointer, I modified database2.py to specify a local (non-network) drive as the location of the metadata.db file as follows: Code: self.dbpath = os.path.join('c:\\calibre\\db', 'metadata.db') The location of the library itself (with all the books) and the directory I was importing from remained on a NAS drive connected via a local gigabit network. The NAS drive is RAID 5 (striping with parity; 4 physical drives) where I get 7 MB/sec sustained transfer rates. The local disk is not RAID, and is actually slower than the NAS in many scenarios (4.5 MB/sec using the same benchmark). For both tests, I started with an empty library (no files, no metadata.db file). I used Calibre 0.5.14 on Windows XP. For the test, I imported a directory tree containing 753 ebooks (mostly .lit). Before the change, it took 54 minutes to import the library. After the change, it took 10 minutes to import the library. In both cases, the vast majority of the time was spent adding the books to the database (after scanning the files and reading the metadata), so I definitely suspect sqllite performance over SMB. I also noticed that the import time does not scale linearly with the number of books (size of the database) when the metadata.db file was on the NAS. The first 20% or so (according to the progress bar) took about the same amount of time in both cases, but the NAS case was really crawling by the end. Yesterday, it took over 4 hours to import 1600 books on the NAS (I'm not sure exactly how long it took because I went to bed), again starting with an empty database. Kovid, are you open to adding an optional config parameter to specify an alternate location for metadata.db for NAS users like us? Thanks, Todd Klaus Last edited by tklaus; 06-11-2009 at 02:02 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Scalability and performance	blither	Calibre	1	08-10-2010 08:31 PM
DR800 Performance	jjansen	iRex	18	08-03-2010 09:10 PM
PERFORMANCE TURNING PAGES	musteman	iRex	21	07-06-2010 10:00 AM
Kobo Desktop Performance	PeterT	Kobo Reader	4	05-07-2010 12:55 PM
Performance in Bulk Metadata Changes	pfooti	Calibre	3	01-02-2010 07:59 PM

06-10-2009, 01:00 AM	#2
itimpi Wizard Posts: 4,552 Karma: 950151 Join Date: Nov 2008 Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)	Not that I know of. I have reverted to having my Calibre files held locally, and then using RichCopy to keep my network copy synchronised with my local copy. The increase in Calibre perfomance is of the order of 100 times when running locally so it is well worth the overheads of running Richcopy periodically. From what I have read elsewhere I suspect that the issue may well be Sqlite performance across networks, particularily running off Samba shares. It could well be worth carrying out s systematic timing exercise to prove if this is the issue.

06-10-2009, 01:12 AM	#3
tklaus Connoisseur Posts: 71 Karma: 131 Join Date: Nov 2007 Location: San Jose, CA Device: iPad, Kindle DX	I've noticed the same thing, and I also have a gigabit network with a NAS accessed via Samba (NAS is Linux). SQLLite sounds like a plausible culprit, is there any way to configure Calibre to store the database file on a different drive? I don't have enough local disk space for all my books, but the database file is small.

06-10-2009, 01:36 AM	#4
tklaus Connoisseur Posts: 71 Karma: 131 Join Date: Nov 2007 Location: San Jose, CA Device: iPad, Kindle DX	Or if someone can point me to a location in the source where I can hack an alternate path to the metadata.db file, I'll try this experiment myself.

06-10-2009, 05:13 PM	#6
tklaus Connoisseur Posts: 71 Karma: 131 Join Date: Nov 2007 Location: San Jose, CA Device: iPad, Kindle DX	Thanks, I'll try that out and report back.

06-10-2009, 08:04 PM	#7
tklaus Connoisseur Posts: 71 Karma: 131 Join Date: Nov 2007 Location: San Jose, CA Device: iPad, Kindle DX	One more question (I'm new to python), where does the output of the print statements go? Is there a log file somewhere, or is there a command-line flag to enable more verbose output? I'm using the binary distribution and I used calibre-debug to replace database2.py with my modified version. Thanks! p.s., I love Calibre, it's a great program, plus it gives me an excuse to learn Python, which has been on my todo list for a while....

06-10-2009, 08:32 PM	#8
kovidgoyal creator of calibre Posts: 43,858 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	If you're on windows run the GUI as calibre-debug -g from a command prompt Then the output of the print statements will show up in the command prompt.

06-11-2009, 02:14 AM	#10
kovidgoyal creator of calibre Posts: 43,858 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Hmm open a ticket and I'll add a config option (you'll have to edit the calibre config files by hand to set it as I'm not going to implement a GUI for it).

06-11-2009, 03:33 AM	#11
tklaus Connoisseur Posts: 71 Karma: 131 Join Date: Nov 2007 Location: San Jose, CA Device: iPad, Kindle DX	Done, thanks! Ticket #2575. None of the listed components seemed appropriate, so I left it as 'Graphical User Interface'

07-30-2009, 12:45 AM	#12
elfchief Junior Member Posts: 2 Karma: 10 Join Date: Jan 2008 Device: Kindle	Is there any ETA on having this implemented? It looks like there's a config file option for it now, but looking at the code, it seems to ignore it right now. I, too, have discovered and tracked down the horrible, horrible performance of sqlite over smb, and it's making Calibre almost useless to me right now. I'd love to actually be able to, y'know, use it. Willing to beg if needed!

Advert

Advert