Poor performance with large library

[BartZorn]

I have not been using calibre for a while, but since maybe version 3.2, the database performance has severely degraded with large databases.

Previously, the size of the database did not seem to be important, but now I have to wait 5 minutes or more for a single change.

OK, we have a ridiculous large database, over 30000 entries.

I am running calibre 3.7, on a Core i7 laptop with plenty memory, The database resides on a file server with a Gb ethernet connection.

With an (almost) empty database, performance is excellent.

Of course I am willing to provide more info if requested.

TIA,

Bart

[PeterT]

Why on earth do people persist in placing calibre libraries on network drives....

[JSWolf]

Originally Posted by PeterT

Why on earth do people persist in placing calibre libraries on network drives....

I do as well. But I don't have nearly as many books.

[itimpi]

Originally Posted by BartZorn

OK, we have a ridiculous large database, over 30000 entries.

. That is not a particularly large library. I have over 45000 in mine and Performance is fine, but all files are on a local HDD.

I am running calibre 3.7, on a Core i7 laptop with plenty memory, The database resides on a file server with a Gb ethernet connection.

This will be the cause of the performance issue. Database performance over a network is notoriously bad. Also it is not an officially supported way of running - only running off a local drive is officially supported.

I used to run with the Books in the Calibre library on my NAS and the database on a local drive. That gives nearly as good performance as having everything local. I would then periodically sync the database back to the Calibre library on the NAS.

[BetterRed]

@BartZorn - see following sections of the Calibre User Manual Ψ²

Frequently Asked Questions ==>> Do not put your calibre library on a networked drive.

Customising Calibre ==>> Environment variables - specifically CALIBRE_OVERRIDE_DATABASE_PATH

BR

[BartZorn]

OK, I did a test. I am absolutely sure that nobody else is using the library.
Calibre is version 3.7, both on the fileserver and my laptop.

Startup time on the server (Windows Server 2016): 4:18
Startup time on my laptop (Windows 10 Pro) : 2:36

To me it seems that a library on a network drive is faster than a local drive.

Anyway, I will have a look at the other tips, and also at the content server solution.

Thanks,

Bart

[mbovenka]

Originally Posted by BartZorn

Startup time on the server (Windows Server 2016): 4:18
Startup time on my laptop (Windows 10 Pro) : 2:36

And both are quite slow.

My library is far larger than yours (230K entries, metadata.db alone is over 500MB) and Calibre 3.7 64bit starts in about 40 seconds.

This on a Win10-64 Home, Core i7 laptop with SATA HD (no SSD).

[PeterT]

One thing we've seen in the past is slow downs caused by custom column definitions.

Do you have any custom columns defined, and if so would you like to share their definitions?

[JSWolf]

Roughly 30,000 entries all doing calculations for custom columns can slow things down locally or networked.

[BartZorn]

Originally Posted by PeterT

One thing we've seen in the past is slow downs caused by custom column definitions.

Do you have any custom columns defined, and if so would you like to share their definitions?

I don't use any custom column definitions.

Bart

[uli]

When "testing" my backup over the network, I've seen poor performance in later versions Calibre and FreeNAS. Slow open, several minutes to edit metadata, can't delete, etc.

Probably bad Samba defaults. For example I switched to Turnkey Linux Filesever and have no more issues.

[JSWolf]

Originally Posted by uli

When "testing" my backup over the network, I've seen poor performance in later versions Calibre and FreeNAS. Slow open, several minutes to edit metadata, can't delete, etc.

Probably bad Samba defaults. For example I switched to Turnkey Linux Filesever and have no more issues.

Version 3.x takes longer to run than version 2.x. That's not including the time to load the database. That seems to me to be about the same.

[BetterRed]

@Bartzorn - Have you tried running calibre with no optional plugins - calibre --ignore-plugins?

How big is the database (metadata.db)?

BR

[BartZorn]

metadat.db is about 61 MB.

Ignoring plugins did not make a noticeable difference. Initializing User Interface takes by far the longest time.

Bart

[BetterRed]

Originally Posted by BartZorn

metadat.db is about 61 MB.

Ignoring plugins did not make a noticeable difference. Initializing User Interface takes by far the longest time.

That database size, 61MB for 30,000 books, doesn't seem abnormal to me, others may care to comment.

Could you run calibre at the server in debug mode and post the log here?

That it is so slow when run from server suggests the server itself is the bottleneck, rather that anything related to the network.

What file system is the server running (NTFS or ??), how much memory does it have, and what other applications run at the server?

Have you tried relocating the database, as suggested in posts 3 and 4.

BR