Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 06-10-2009, 01:19 AM   #1
mrwindex
Junior Member
mrwindex began at the beginning.
 
Posts: 1
Karma: 10
Join Date: Jun 2009
Device: iLiad V2
Calibre performance over the network

Hi,

I have read in the past about some users having difficulties with calibre and poor network performance. I am having this problem myself.
The last beta version is definitely not addressing any of these issues.
Calibre is having extremely low performance and most of the time over 60 seconds delay to tag a file with ISBN and cover data.


Is there a "fix" or workaround for these issues?



Using:
Calibre 0.6.0b2, before 0.5.14
Windows XP SP3
AVG Internet Security 8.0
hp xw8400 dual Xeon 3.2 Ghz 4Gb ram

EX487 with gigabit Ethernet
Library size: about 6000 pdf and djvu files
mrwindex is offline   Reply With Quote
Old 06-10-2009, 02:00 AM   #2
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,539
Karma: 950151
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
Not that I know of.

I have reverted to having my Calibre files held locally, and then using RichCopy to keep my network copy synchronised with my local copy. The increase in Calibre perfomance is of the order of 100 times when running locally so it is well worth the overheads of running Richcopy periodically.

From what I have read elsewhere I suspect that the issue may well be Sqlite performance across networks, particularily running off Samba shares. It could well be worth carrying out s systematic timing exercise to prove if this is the issue.
itimpi is offline   Reply With Quote
Advert
Old 06-10-2009, 02:12 AM   #3
tklaus
Connoisseur
tklaus doesn't littertklaus doesn't litter
 
tklaus's Avatar
 
Posts: 71
Karma: 131
Join Date: Nov 2007
Location: San Jose, CA
Device: iPad, Kindle DX
I've noticed the same thing, and I also have a gigabit network with a NAS accessed via Samba (NAS is Linux).

SQLLite sounds like a plausible culprit, is there any way to configure Calibre to store the database file on a different drive? I don't have enough local disk space for all my books, but the database file is small.
tklaus is offline   Reply With Quote
Old 06-10-2009, 02:36 AM   #4
tklaus
Connoisseur
tklaus doesn't littertklaus doesn't litter
 
tklaus's Avatar
 
Posts: 71
Karma: 131
Join Date: Nov 2007
Location: San Jose, CA
Device: iPad, Kindle DX
Or if someone can point me to a location in the source where I can hack an alternate path to the metadata.db file, I'll try this experiment myself.
tklaus is offline   Reply With Quote
Old 06-10-2009, 05:18 AM   #5
gwynevans
Wizzard
gwynevans ought to be getting tired of karma fortunes by now.gwynevans ought to be getting tired of karma fortunes by now.gwynevans ought to be getting tired of karma fortunes by now.gwynevans ought to be getting tired of karma fortunes by now.gwynevans ought to be getting tired of karma fortunes by now.gwynevans ought to be getting tired of karma fortunes by now.gwynevans ought to be getting tired of karma fortunes by now.gwynevans ought to be getting tired of karma fortunes by now.gwynevans ought to be getting tired of karma fortunes by now.gwynevans ought to be getting tired of karma fortunes by now.gwynevans ought to be getting tired of karma fortunes by now.
 
gwynevans's Avatar
 
Posts: 1,402
Karma: 2000000
Join Date: Nov 2007
Location: UK
Device: iPad 2, iPhone 6s, Kindle Voyage & Kindle PaperWhite
Quote:
Originally Posted by tklaus View Post
Or if someone can point me to a location in the source where I can hack an alternate path to the metadata.db file, I'll try this experiment myself.
Probably the line "self.dbpath = os.path.join(library_path, 'metadata.db')" in "LibraryDatabase2.__init__" in "calibre.library.database2.py" is where you need to start. Try commenting that out & setting an absolute path to your local 'metadata.db' & see if that works/makes any difference.

To test, you'll either need to build calibre fully from source (best for debugging, etc) or failing that, get the source that matches your released version and use calibre-debug to replace the frozen module (in library.zip - take your own backup ) in the distribution.
gwynevans is offline   Reply With Quote
Advert
Old 06-10-2009, 06:13 PM   #6
tklaus
Connoisseur
tklaus doesn't littertklaus doesn't litter
 
tklaus's Avatar
 
Posts: 71
Karma: 131
Join Date: Nov 2007
Location: San Jose, CA
Device: iPad, Kindle DX
Thanks, I'll try that out and report back.
tklaus is offline   Reply With Quote
Old 06-10-2009, 09:04 PM   #7
tklaus
Connoisseur
tklaus doesn't littertklaus doesn't litter
 
tklaus's Avatar
 
Posts: 71
Karma: 131
Join Date: Nov 2007
Location: San Jose, CA
Device: iPad, Kindle DX
One more question (I'm new to python), where does the output of the print statements go? Is there a log file somewhere, or is there a command-line flag to enable more verbose output? I'm using the binary distribution and I used calibre-debug to replace database2.py with my modified version.

Thanks!

p.s., I love Calibre, it's a great program, plus it gives me an excuse to learn Python, which has been on my todo list for a while....
tklaus is offline   Reply With Quote
Old 06-10-2009, 09:32 PM   #8
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 40,299
Karma: 18000932
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
If you're on windows run the GUI as

calibre-debug -g

from a command prompt

Then the output of the print statements will show up in the command prompt.
kovidgoyal is offline   Reply With Quote
Old 06-11-2009, 02:51 AM   #9
tklaus
Connoisseur
tklaus doesn't littertklaus doesn't litter
 
tklaus's Avatar
 
Posts: 71
Karma: 131
Join Date: Nov 2007
Location: San Jose, CA
Device: iPad, Kindle DX
With gwynevans pointer, I modified database2.py to specify a local (non-network) drive as the location of the metadata.db file as follows:

Code:
self.dbpath = os.path.join('c:\\calibre\\db', 'metadata.db')
The location of the library itself (with all the books) and the directory I was importing from remained on a NAS drive connected via a local gigabit network. The NAS drive is RAID 5 (striping with parity; 4 physical drives) where I get 7 MB/sec sustained transfer rates. The local disk is not RAID, and is actually slower than the NAS in many scenarios (4.5 MB/sec using the same benchmark).

For both tests, I started with an empty library (no files, no metadata.db file). I used Calibre 0.5.14 on Windows XP. For the test, I imported a directory tree containing 753 ebooks (mostly .lit).

Before the change, it took 54 minutes to import the library.
After the change, it took 10 minutes to import the library.

In both cases, the vast majority of the time was spent adding the books to the database (after scanning the files and reading the metadata), so I definitely suspect sqllite performance over SMB. I also noticed that the import time does not scale linearly with the number of books (size of the database) when the metadata.db file was on the NAS. The first 20% or so (according to the progress bar) took about the same amount of time in both cases, but the NAS case was really crawling by the end. Yesterday, it took over 4 hours to import 1600 books on the NAS (I'm not sure exactly how long it took because I went to bed), again starting with an empty database.

Kovid, are you open to adding an optional config parameter to specify an alternate location for metadata.db for NAS users like us?

Thanks,
Todd Klaus

Last edited by tklaus; 06-11-2009 at 03:02 AM.
tklaus is offline   Reply With Quote
Old 06-11-2009, 03:14 AM   #10
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 40,299
Karma: 18000932
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Hmm open a ticket and I'll add a config option (you'll have to edit the calibre config files by hand to set it as I'm not going to implement a GUI for it).
kovidgoyal is offline   Reply With Quote
Old 06-11-2009, 04:33 AM   #11
tklaus
Connoisseur
tklaus doesn't littertklaus doesn't litter
 
tklaus's Avatar
 
Posts: 71
Karma: 131
Join Date: Nov 2007
Location: San Jose, CA
Device: iPad, Kindle DX
Done, thanks! Ticket #2575. None of the listed components seemed appropriate, so I left it as 'Graphical User Interface'
tklaus is offline   Reply With Quote
Old 07-30-2009, 01:45 AM   #12
elfchief
Junior Member
elfchief began at the beginning.
 
Posts: 2
Karma: 10
Join Date: Jan 2008
Device: Kindle
Is there any ETA on having this implemented? It looks like there's a config file option for it now, but looking at the code, it seems to ignore it right now.

I, too, have discovered and tracked down the horrible, horrible performance of sqlite over smb, and it's making Calibre almost useless to me right now. I'd love to actually be able to, y'know, use it.

Willing to beg if needed!
elfchief is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Scalability and performance blither Calibre 1 08-10-2010 09:31 PM
DR800 Performance jjansen iRex 18 08-03-2010 10:10 PM
PERFORMANCE TURNING PAGES musteman iRex 21 07-06-2010 11:00 AM
Kobo Desktop Performance PeterT Kobo Reader 4 05-07-2010 01:55 PM
Performance in Bulk Metadata Changes pfooti Calibre 3 01-02-2010 08:59 PM


All times are GMT -4. The time now is 10:29 PM.


MobileRead.com is a privately owned, operated and funded community.