Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 09-06-2017, 02:41 PM   #1
BartZorn
Member
BartZorn began at the beginning.
 
Posts: 19
Karma: 10
Join Date: Jul 2012
Device: Sony PRS-T1
Poor performance with large library

I have not been using calibre for a while, but since maybe version 3.2, the database performance has severely degraded with large databases.

Previously, the size of the database did not seem to be important, but now I have to wait 5 minutes or more for a single change.

OK, we have a ridiculous large database, over 30000 entries.

I am running calibre 3.7, on a Core i7 laptop with plenty memory, The database resides on a file server with a Gb ethernet connection.

With an (almost) empty database, performance is excellent.

Of course I am willing to provide more info if requested.

TIA,

Bart
BartZorn is offline   Reply With Quote
Advert
Old 09-06-2017, 05:02 PM   #2
PeterT
Grand Sorcerer
PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.
 
PeterT's Avatar
 
Posts: 9,969
Karma: 58567737
Join Date: Nov 2007
Location: Toronto
Device: Wife: Touch, Arc, Vox Me: Nexus 7, Glo
Why on earth do people persist in placing calibre libraries on network drives....
PeterT is offline   Reply With Quote
Old 09-06-2017, 05:11 PM   #3
JSWolf
Resident Curmudgeon
JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.
 
JSWolf's Avatar
 
Posts: 49,064
Karma: 40053823
Join Date: Nov 2006
Location: Roslindale, Massachusetts
Device: Kobo Aura H2O, Sony PRS-650, Sony PRS-T1, nook STR, iPad 4, iPhone 5
Quote:
Originally Posted by PeterT View Post
Why on earth do people persist in placing calibre libraries on network drives....
I do as well. But I don't have nearly as many books.
JSWolf is offline   Reply With Quote
Old 09-06-2017, 06:11 PM   #4
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,368
Karma: 910965
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
Quote:
Originally Posted by BartZorn View Post
OK, we have a ridiculous large database, over 30000 entries.
. That is not a particularly large library. I have over 45000 in mine and Performance is fine, but all files are on a local HDD.
Quote:
I am running calibre 3.7, on a Core i7 laptop with plenty memory, The database resides on a file server with a Gb ethernet connection.
This will be the cause of the performance issue. Database performance over a network is notoriously bad. Also it is not an officially supported way of running - only running off a local drive is officially supported.

I used to run with the Books in the Calibre library on my NAS and the database on a local drive. That gives nearly as good performance as having everything local. I would then periodically sync the database back to the Calibre library on the NAS.
itimpi is offline   Reply With Quote
Old 09-06-2017, 07:04 PM   #5
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 8,972
Karma: 7214889
Join Date: Mar 2012
Location: Sydney Australia
Device: none
@BartZorn - see following sections of the Calibre User Manual Ψ²

Frequently Asked Questions ==>> Do not put your calibre library on a networked drive.

Customising Calibre ==>> Environment variables - specifically CALIBRE_OVERRIDE_DATABASE_PATH

BR
BetterRed is online now   Reply With Quote
Advert
Old 09-07-2017, 06:32 AM   #6
BartZorn
Member
BartZorn began at the beginning.
 
Posts: 19
Karma: 10
Join Date: Jul 2012
Device: Sony PRS-T1
OK, I did a test. I am absolutely sure that nobody else is using the library.
Calibre is version 3.7, both on the fileserver and my laptop.

Startup time on the server (Windows Server 2016): 4:18
Startup time on my laptop (Windows 10 Pro) : 2:36

To me it seems that a library on a network drive is faster than a local drive.

Anyway, I will have a look at the other tips, and also at the content server solution.

Thanks,

Bart
BartZorn is offline   Reply With Quote
Old 09-07-2017, 09:59 AM   #7
mbovenka
Wizard
mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.
 
Posts: 1,640
Karma: 6400817
Join Date: Oct 2007
Location: Almere, The Netherlands
Device: Kobo Aura ONE
Quote:
Originally Posted by BartZorn View Post
Startup time on the server (Windows Server 2016): 4:18
Startup time on my laptop (Windows 10 Pro) : 2:36
And both are quite slow.

My library is far larger than yours (230K entries, metadata.db alone is over 500MB) and Calibre 3.7 64bit starts in about 40 seconds.

This on a Win10-64 Home, Core i7 laptop with SATA HD (no SSD).
mbovenka is offline   Reply With Quote
Old 09-07-2017, 11:11 AM   #8
PeterT
Grand Sorcerer
PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.
 
PeterT's Avatar
 
Posts: 9,969
Karma: 58567737
Join Date: Nov 2007
Location: Toronto
Device: Wife: Touch, Arc, Vox Me: Nexus 7, Glo
One thing we've seen in the past is slow downs caused by custom column definitions.

Do you have any custom columns defined, and if so would you like to share their definitions?
PeterT is offline   Reply With Quote
Old 09-07-2017, 11:38 AM   #9
JSWolf
Resident Curmudgeon
JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.
 
JSWolf's Avatar
 
Posts: 49,064
Karma: 40053823
Join Date: Nov 2006
Location: Roslindale, Massachusetts
Device: Kobo Aura H2O, Sony PRS-650, Sony PRS-T1, nook STR, iPad 4, iPhone 5
Roughly 30,000 entries all doing calculations for custom columns can slow things down locally or networked.
JSWolf is offline   Reply With Quote
Old 09-07-2017, 12:28 PM   #10
BartZorn
Member
BartZorn began at the beginning.
 
Posts: 19
Karma: 10
Join Date: Jul 2012
Device: Sony PRS-T1
Quote:
Originally Posted by PeterT View Post
One thing we've seen in the past is slow downs caused by custom column definitions.

Do you have any custom columns defined, and if so would you like to share their definitions?
I don't use any custom column definitions.

Bart
BartZorn is offline   Reply With Quote
Old 09-07-2017, 01:56 PM   #11
uli
Enthusiast
uli has a complete set of Star Wars action figures.uli has a complete set of Star Wars action figures.uli has a complete set of Star Wars action figures.
 
Posts: 29
Karma: 296
Join Date: Jun 2011
Location: Montreal
Device: Kobo, Calibre
When "testing" my backup over the network, I've seen poor performance in later versions Calibre and FreeNAS. Slow open, several minutes to edit metadata, can't delete, etc.

Probably bad Samba defaults. For example I switched to Turnkey Linux Filesever and have no more issues.
uli is offline   Reply With Quote
Old 09-07-2017, 04:40 PM   #12
JSWolf
Resident Curmudgeon
JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.
 
JSWolf's Avatar
 
Posts: 49,064
Karma: 40053823
Join Date: Nov 2006
Location: Roslindale, Massachusetts
Device: Kobo Aura H2O, Sony PRS-650, Sony PRS-T1, nook STR, iPad 4, iPhone 5
Quote:
Originally Posted by uli View Post
When "testing" my backup over the network, I've seen poor performance in later versions Calibre and FreeNAS. Slow open, several minutes to edit metadata, can't delete, etc.

Probably bad Samba defaults. For example I switched to Turnkey Linux Filesever and have no more issues.
Version 3.x takes longer to run than version 2.x. That's not including the time to load the database. That seems to me to be about the same.
JSWolf is offline   Reply With Quote
Old 09-07-2017, 04:55 PM   #13
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 8,972
Karma: 7214889
Join Date: Mar 2012
Location: Sydney Australia
Device: none
@Bartzorn - Have you tried running calibre with no optional plugins - calibre --ignore-plugins?

How big is the database (metadata.db)?

BR
BetterRed is online now   Reply With Quote
Old 09-08-2017, 03:56 AM   #14
BartZorn
Member
BartZorn began at the beginning.
 
Posts: 19
Karma: 10
Join Date: Jul 2012
Device: Sony PRS-T1
metadat.db is about 61 MB.

Ignoring plugins did not make a noticeable difference. Initializing User Interface takes by far the longest time.

Bart
BartZorn is offline   Reply With Quote
Old 09-08-2017, 06:29 AM   #15
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 8,972
Karma: 7214889
Join Date: Mar 2012
Location: Sydney Australia
Device: none
Quote:
Originally Posted by BartZorn View Post
metadat.db is about 61 MB.

Ignoring plugins did not make a noticeable difference. Initializing User Interface takes by far the longest time.
That database size, 61MB for 30,000 books, doesn't seem abnormal to me, others may care to comment.

Could you run calibre at the server in debug mode and post the log here?

That it is so slow when run from server suggests the server itself is the bottleneck, rather that anything related to the network.

What file system is the server running (NTFS or ??), how much memory does it have, and what other applications run at the server?

Have you tried relocating the database, as suggested in posts 3 and 4.

BR
BetterRed is online now   Reply With Quote
Reply

Tags
performance

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Large Library Performance Comparisons Down South Calibre 9 06-08-2013 11:37 AM
Performance issue with large library audeojude Calibre 25 12-07-2011 04:44 PM
DX performance on large (>100MB) pdf files? Blackguard Amazon Kindle 6 06-16-2009 06:57 PM
Library size and performance MrFinch Calibre 4 01-01-2009 04:11 AM
Performance with large Libraries? itimpi Calibre 3 12-14-2008 02:46 PM


All times are GMT -4. The time now is 09:12 PM.


MobileRead.com is a privately owned, operated and funded community.