Calibre - running more slowly

[Joanna]

Recently, I've made quite a revolution in my Calibre libraries, changing its organization, adding books, merging libraries, adding new settings etc.

I'm very happy about my new Calibre library and at the same time, quite sad, as the performance seems to be worse than before. Calibre is less 'responsive', scrolling is more 'jerky', it takes a little bit longer to edit the column in the table view etc. It's not terrible but got me worried.

The changes I've introduced recently:
- merging several libraries - before, the biggest one had ca. 1400 books, now I have almost 2300,
- using virtual library tabs,
- increased number of custom columns - partly because some of them existed in one of my libraries only. There are 23 custom columns at the moment, most of them set only for a relatively small subset of books. Only one of them is built from other columns,
- having used yet another custom column with 'program: user_categories()' template (but I've removed that one, as it's supposed to be heavy on resources),
- increased number of categories displayed in the Tag Browser (21 in total),
- increased number of 'big' books - ca. 100 over 50 MB, mostly .zip due to the fact I've added audiobooks and language courses to the library,
- changing rules for column icons and, to a lesser extent, emblems (more rules - ca. 20 for columns icons, a bit more complex than before).

I usually have no more than 10 columns displayed in the table view at the same time, and tend to work with subsets of books (virtual libraries, under 1000 books). In the book details, no more than 10 columns are shown for any given book (and usually, there are fewer of them). I don't like hiding the Tag Browser.

As I'm really happy about my setup, I would prefer not to split the library back into several physical ones.

What do you think about it? Any way to improve the performance without a new revolution?

Using Calibre Portable 2.44 on an external HDD, Windows 7 Professional 64-bit, the laptop is on the slower side (4 GB RAM, SSD, old processor), usually a couple of programs running at the same time.

[eschwartz]

The size of a book is irrelevant, since all calibre has is a reference that a file is there (plus a one-time recorded size).
And the number of records is not one of the big slowdown factors -- some people have many many thousands of books without being noticeably slowed down.

Displaying many columns in the Tag Browser is one thing that can slow calibre down.
Fortunately you say you only have one custom column created from other columns -- those are also heavy since they are dynamically calculated.
Might also want to see if you can trim down the column icons.

...

Most of these things are all right in moderation -- and useful -- but the thing is, they build up and cumulatively slow calibre down.

[Joanna]

I've hidden most of the columns shown in the Tag Browser. It looks and feels strange but I'll see whether that helps (and then try re-adding some of them gradually). I'm waiting for the new Calibre release where columns with no items will be hidden automatically in the Tag Browser (as many of the columns are relevant only for one of my virtual libraries).

As far as the column icons are concerned: I have 6 simple rules (mostly with one condition only) for the Title column.

The things get more problematic as far as the icons in the Languages column are concerned. For each book, a flag is displayed, depending on the language of the book. If the book is a language learning material, the flag depends on the content of the custom #target column. For some languages not included in the rules, a generic flag is displayed.

I'm really fond of this solution but the truth is it does require lots of rules (17 in total at the moment). Do you think this might be the biggest problem? (Before I introduced the #target column, I'd only had 7 or 8 rules for the flag.) I know, I know, it's not about finding one culprit - but do you think this still qualifies as "in moderation"?

(I also have 14 rules for column coloring, but here I could pretty easily get rid off some of them.)

[eschwartz]

17 rules is definitely a lot. I'm afraid I haven't tested the comparative effects of custom columns, column icons, columns shown in the tag browser, etc. though -- I only know that all of those things can have effects.

I don't use column icons at all.

[DaltonST]

@Joanna:

These might help and won't hurt:

* 'Vacuum' your Calibre Library metadata.db file by using the Calibre Menu Option: Library Maintenance > Check Library.

* Compress/Defragment your Calibre Library files using a Windows File Defragmenter, such as http://www.piriform.com/defraggler/download .

* Compress/Defragment the external HDD on which your Calibre Portable 2.44 resides.


DaltonST

[Divingduck]

@Joanna: In addition check, if AV is slowing down your library.

[fidvo]

I suspect your custom rules might be causing the slowdown.

The only "rule" I have is a basic column coloring rule, and I noticed a slight jerkiness when scrolling through books since I added it, although I don't know if that's the culprit because I made several other modifications to my library at the same time.

If the number of items in the Tag Browser is the problem, you might consider trying to find a way to consolidate your columns. I've found that a single tag-like field set up as a hierarchy can replace several fields; just use the field name as the top level of the hierarchy. e.g. instead of a "Genre" field with the value "Mystery" in it, you could have a field with the value "Genre.Mystery".

Likewise, if you have several Yes/No columns with mutually exclusive values, you could consolidate them into a single field with a fixed set of permitted values. If the values aren't mutually exclusive, you could still turn them into a single tag-like column.

[DaltonST]

@Joanna:

- using virtual library tabs
- increased number of custom columns - partly because some of them existed in one of my libraries only. There are 23 custom columns at the moment, most of them set only for a relatively small subset of books. - increased number of custom columns - partly because some of them existed in one of my libraries only. There are 23 custom columns at the moment, most of them set only for a relatively small subset of books.

What I do is use 'View Manager' Views based on what columns (standard and custom) that I want to see. I have certain Virtual Libraries that have unique Views for that reason.

The "relatively small subset of books" deserve a VL of their own, along with a matching View.

No other Views need to show those custom columns applicable to that "relatively small subset of books".

No View needs to show any standard column that you don't wish to see either.

Obviously, you will also need a View that "shows everything".



DaltonST

[chaley]

FWIW: standard custom columns have almost no effect on calibre performance. Their values are read into memory when calibre starts and no further computation is needed unless their value is changed. The tag browser is completely computed when something changes using in-memory values, so the number of custom columns has no effect on scrolling. The auto-hide feature will have no impact at all, because the value must be computed in order to know that there aren't any values.

If I had to guess, it would say that it is the icon rules. Each rule is converted to a template, the equivalent of a column built from other columns (a "composite" column), so 17 rules will have a similar performance impact as 17 composite columns. You might be able to make it run faster by using a single advanced icon rule to compute the list of icons to apply to a row. Even better would be to use a custom template function to compute the list of icons. If you use "composed rules" then you "pay" for evaluating each template in the composition set.

Another possibility is memory. Calibre uses an in-memory copy of its database. If there isn't enough memory for that copy plus all the other data structures (tag browser, cover cache, template cache, etc) then your machine will start using disk-based virtual memory, which is very slow.

[Joanna]

Thanks for your feedback. I've started defragmenting (it has turned out that the main db file was quite heavily fragmented, and quite many .opf files are fragmented, too) and we'll see (for now, I've closed Calibre).

One question about rules: is there any method to disable/enable them instead of removing them and setting up once again?

// EDIT
@Chaley, as the 17 rules are used to display only one type of icons in one column, I guess it has to be possible to replace them by an advanced/composed rule. Thanks for the tip! Now I just have to understand how advanced icon rules/custom templates work .

@DaltonST, that's basically what I do (with views and VLs) - a custom view for most of VLs, and irrelevant columns are hidden in other VLs.
Thanks for the tip about defragmenting!

[chaley]

Originally Posted by Joanna

One question about rules: is there any method to disable/enable them instead of removing them and setting up once again?

Nope.

[BetterRed]

@Joanna - I have a library with 6 icon only columns, 3 are based on whether the underlying data is empty or not (i.e one rule), the others have 3 or 4 simple rules. Number of books is 74,000. some of them contain +300MB video files. I do all my metadata editing in the book list table, I've not noticed this library is substantially slower than smaller less complex libraries.

Do DaltonST's suggestions to vacuum the database and defrag the library files.

Try hiding the Tag Browser - Shift+Alt+T to toggle

You can use the View Manager PI to help determine if certain columns are the problem. Create a new view that only includes the 'standard' columns you want, assuming that's quick (if it's not then the cause presumably lies 'outside' the library), and then select additional columns progressively, I'd start with the 17 rule icon.

I have a vague memory that colouring individual cells (rather than the whole row) effected performance - my guess is that would have been pre version 1.0, so it may not be true today.

When you edit metadata in the book list I believe every cell you edit writes a new metadata.opf file, the writes are done via a queue; so, in and of themselves they don't create delays, but if your AV were to lock and scan the file every time it was written then that may effect performance. But that would not explain why scrolling through the book list is slow.

BR

[theducks]

A calculated custom column is calculated whether it shows or not.

Can you calculate just Once (in a while)?

Use bulk mode Search and replace to populate a STATIC custom column
( remember to save your search )

[Joanna]

And what about 'calculated' icons?

If a column is not shown in a given view, will the icons still be computed? I could check what the performance is like with my 17 rule icon column hidden (I have icons in two columns only, and both columns are shown in every view I use, so I haven't had any comparison).

[chaley]

Originally Posted by theducks

A calculated custom column is calculated whether it shows or not.

A composite column is evaluated when its value is used. A composite column is used when it is displayed, shown in the tag browser, used to sort the book list, or used as a value in some other composite column/template/rule.

Originally Posted by Joanna

And what about 'calculated' icons?

If a column is not shown in a given view, will the icons still be computed? I could check what the performance is like with my 17 rule icon column hidden (I have icons in two columns only, and both columns are shown in every view I use, so I haven't had any comparison).

No, they are not evaluated. Icon and color rules are evaluated when the cell to which the rules are attached is displayed. If you don't display the cell then the rule is not evaluated. This is why horizontal scrolling can sometimes stutter; the icon/color rules for the newly-displayed cells are being evaluated.