Calibre slowness as library expands

[microcity]

I started using Calibre about a year ago, and when my library was 10 volumes it was great. Gradually my library has grown to more than 10K volumes, and Calibre has become almost unusably slow. Startup time is more than a minute, and even basic searches can take more than 30 seconds. Does anyone have suggestions on how I can accelerate Calibre?

[kiwidude]

Welcome to MobileRead.

Preferences -> Searching -> tick the "Limit the searched metadata" box. Makes an enormous difference to search speed.

There are a number of threads on this subject in the forum if you do a search.

[mbovenka]

Originally Posted by kiwidude

Preferences -> Searching -> tick the "Limit the searched metadata" box. Makes an enormous difference to search speed.

Or train yourself so use the search specifiers, like 'author:' and 'title:'. That helps a bunch too.

[unboggling]

Speeding Up:

1. In external security software, disable virus scanning of official calibre Library folders.

2. Hide the Tag Browser in calibre when not using it directly, particularly before quitting calibre so it won't be open the next time calibre starts up.

3. Disable unnecessary plugins where calibre allows, such as Device Interface plugins for devices you don't use, and any installed but currently unused third-party plugins such as for Metadata Source and User Interface. All enabled plugins take time to initialize at calibre startup. You can enable rarely-used plugins specifically only when you need them.

4. Use as few columns "built from other columns" as possible.

5. Avoid using complex template language or programming particularly where it has to iterate through large numbers of books in the library or restriction, one instance of which occurs at calibre startup.

6. Prefix all searches with the column lookup name, including in saved searches and restrictions. Unprefixed searches in a large library can be time consuming even with Prefs/Searching/"Limit the searched metadata" set to exclude Comments and to include the most useful and simple columns. Also where possible, do intensive search operations within pre-set restrictions ( -- restrictions limit the number of books to be searched through to a subset of the library).

There are probably other performance issues/tips for large libraries that siipped my mind at the moment, or that I haven't learned yet.

[microcity]

unboggling... great suggestions! Turning off the plugins was very annoying in the UI, since you have to click through each one. However, the effects are dramatic: program startup is back to a few seconds, and searches are speedy again. Thanks again!

[unboggling]

You're welcome.

Another thing I do is keep my data clean. Metadata Download often brings in various unstandardized data, for example for Publisher various books may have Random House Inc, Random House Inc., Random House for Young Readers, Random House ThisOrThat. Every several months I go through the tag browser cleaning up the variations to one standard name such as "Random House". I do that for all columns where there may be insignificant variations. I don't know if that speeds up searches or if it's just me being obsessive/compulsive.

[kiwidude]

Just to kill off some urban myths here before they perpetuate...

Disabling plugins will have virtually zero impact on startup time (it will at best be a difference measured in milliseconds, certainly not seconds). I have an 11K book library, twenty-something of my own plugins installed and enabled plus meme's Kindle Collections plugin, and all of the standard calibre plugins enabled. My library starts up in 5 seconds.

Secondly plugins will make zero difference to search time - they are not in the slightest bit involved in that process. The best thing you can do for search performance is twofold. Limiting the search terms advice I gave in my previous post above will make enormous difference if your habit is to just type your search words into the search bar without scope prefixes. Alternatively the advice from unboggling of using search prefixes is for optimum results.

I wouldn't think that rationalising your publishers will make any worthwhile difference to search times (particularly for the enormous ongoing and manual effort involved) though I agree for the OCD types it might be a warm fuzzy.

@microcity - if you have found that your library has sped up, I just don't believe plugins were the reason, it will have been something else that has changed. It could be a an OS update, an AV update, a calibre update, a custom column, a disk problem - but I can all but guarantee it will not have been a plugin as the cause.

[unboggling]

Huh. Thanks for the clarification, kiwidude. I do have a bad habit of operating under insidious bad assumptions. But now I'll not worry about disabling plugins and keeping my data so clean.

[kiwidude]

No probs. Speaking from personal experience elsewhere I know that when you have a problem and you start googling for solutions, you can end up grasping at any suggestion you come across in the hope it is the magic answer you were looking for. I would have hated to see future users come across this thread and draw incorrect conclusions like "plugins will slow down calibre so they are bad".

Startup time, search performance and metadata download are three discrete and entirely separate software tasks. They all have individual solutions to resolve any problems with. Disabling slower metadata download plugins like Overdrive (if you had it enabled which is not the default) is a good suggestion to increase the metadata download performance but it will make no difference to calibre startup or search times.

It would certainly be "possible" to write a GUI plugin that poorly impacts calibre's startup times, but as I said above that is certainly not the case with the plugins I have published/use. On a 3-yr old Win 7 64-bit machine with a not particularly fast hard-drive startup time is around 5 secs for 11K books and about 10 seconds for a 33K book library from memory. I am sure I could make that faster by using an SSD and a newer machine but it is good enough for me - and certainly not the "minutes" that some users come on the forums to report. So if you have a library that is taking considerably longer to load then as the saying goes, "these are not the droids you are looking for..."

[unboggling]

Well, I decided to continue disabling or removing plugins I don't use. Can't hurt.

[Mythlandia]

Disabling the tag browser made a huge difference running my 30k library