Calibre still slow on brand new computers

[BookJunkieLI]

I know this is a topic that frequently comes up but I'm really frustrated. I have two computers that I use, one at home and one at work, for accessing my Calibre libraries on a regular basis. Both computers we're built within the last 3 months. The work one I didn't have a say in what specs were used, but I built the one for home myself. I based my decisions of what components to pick largely on things I'd gleaned from here on the forums in the expectation that Calibre would perform well. That's not the case for the larger of my two libraries.

These are the specs for the computers:
Home
Win 7 Pro 64-bit
AMD Trinity Quad-Core 3.6GHz 100W Desktop APU (CPU+GPU)
8GB (2x4GB) DDR3 SDRAM
Motherboard with SATA 6Gb/s ports
1TB WD Green HDD, 3Gb/s - I admit this is an older drive that I had on hand.

Work
Win 7 Pro 64-bit
Intel Celeron G530 Sandy Bridge Dual-Core 2.4GHz
8GB (2x4GB) DDR3 SDRAM
Motherboard with SATA 3.0GB/s ports
1TB Seagate Barracude HDD, 6.0Gb/s


*Calibre 1.36 Win 64-bit
*Library folders are in Dropbox - synching is paused when I'm doing any metadata editing.
*Main Library has roughly 4.3k books - this works relatively fast
*Import Library has roughly 9.3k books - this one just scrolling up/down or right/left is a lesson in patience. Trying to process single or bulk metadata gives me plenty of time to play a game of Solitaire, sometimes two. Oddly, importing through FFDL doesn't take noticeably longer than usual. I think. I've been importing anywhere from 400-600 files at a time so I usually just let it run while I go off and do other things.
*I don't remember exactly which Plugins I have installed beyond FFDL, Count Pages, and EPUBMerge. I might have Quality Check, Find Duplicates, and maybe Reading List installed. I try to keep it the same across all installations.


Before anyone jumps on Dropbox being the issue I have tried moving the library into My Documents itself with no connection to Dropbox and there was no noticeable increase in speed. I have the library folders listed as exceptions in my virus software so it shouldn't be scanning them, but I have also tried running Calibre with the virus scanner disabled and, again, no noticeable increase in speed.

The Import Library isn't bloated with image files. In fact, because I was curious, the Import Library only takes up 438 MB of space while the Main Library is taking up 1.56 GB of space.

The one thing I am wondering about, is I noticed that the Home computer has 6.0GB/s SATA ports but the HDD is only 3GB/s, and the Work computer is the opposite - 3GB/s SATA ports with a 6.0GB/s HDD. Could that be an issue?

Any help would be appreciated because I know compared to the libraries of others on here my libraries are pretty small. so to find Calibre so bogged down is just incredibly frustrating. If you need any other information let me know. I wasn't sure what would be useful beyond the computer hardware and basic Calibre specs.

[JSWolf]

I don't find Calibre to be slow accessing the libraries. It's not even much of an issue across the network.

[Adoby]

How many custom columns do you have, hidden or visible, and are any of them computed?

The size of the library in bytes doesn't matter. The number of books does, and computed custom columns even more.

When you browse the library using the GUI you are only accessing the database file, not the books themselves.

[BookJunkieLI]

Originally Posted by Adoby

How many custom columns do you have, hidden or visible, and are any of them computed?

The size of the library in bytes doesn't matter. The number of books does, and computed custom columns even more.

When you browse the library using the GUI you are only accessing the database file, not the books themselves.

By computed I assume you mean template columns?

Main Library - 29 Custom Columns, 3 Template Columns
Import Library - 24 Custom Columns, 2 Template Columns

That did get me wondering, would Column Coloring Rules have an affect?
Main Library has 1 Column Color Rule
Import Library has 38 Column Color Rules.
If that is likely the culprit I can toss all but 1 of them without a major issue.

[Adoby]

Yes, I suspect that coloring rules have a very big impact.

Make a backup and remove coloring rules and see what happens. Also check what happens if you remove all custom columns.

[BookJunkieLI]

Originally Posted by Adoby

Yes, I suspect that coloring rules have a very big impact.

Make a backup and remove coloring rules and see what happens. Also check what happens if you remove all custom columns.

Just to see what happened, I deleted the rules in increments. I also realized that one of the custom columns, a non-template, had a color rule. These are the results:

Measurement used is how long it took to scroll right/left.
-No noticeable increase in speed between 16 and 38 color rules
-A small but noticeable increase at 15 color rules
-Larger increase in speed when I removed 3 color rules that had 'has this-but not this' conditions
-At 5 rules speed becomes acceptable with only occasional lag
-No noticeable lag with 1 color rule and 1 custom column w/color rule

Editing speed has increased as well, though I know the number of books I edit at any given time will have some affect on the processing speed. Processing 5 records is not the same as processing 500.

Thanks for the help in figuring this out.
I have to wonder if it might not be worth having a sticky with most common reasons for slowdown in Calibre. It might save a lot of headaches.

[Sabardeyn]

Besides the points you address, having the Tag Browser or Book Details (and possibly the Cover Browser or Cover Grid) displayed while calibre updates the program window, would cause additional delay.

However, I've been away for several months and during that time this issue might have been ameliorated.

[Adoby]

I think the performance was significantly improved with the new database code some months(?) ago, so old posts about ways to improve performance may not be very relevant or even wrong. I haven't had any reason to investigate new ways improve performance, since calibre currently is fast enough for me, without any special tricks.

I have the libraries on a local fast HD and use no custom columns or column coloring rules and only have epubs and pdfs in my main libraries.

[BookJunkieLI]

@Sabardeyn - I believe that's been addressed with the 1.0 update. Using the most ancient computer we have at work - almost 10 yrs old with only 2 GB of RAM, Win XP - I had all 4 of those displayed and experienced no lag in my larger library.

@Adoby - Honestly, I was thinking more along the lines of letting people know that using large numbers of Color Rules, Template Columns, and possibly the Column Icon Rules will cause a slowdown. These are things that appear to constantly require re-calculating even just for scrolling left/right up/down on the main screen.

[theducks]

I mostly do my cleanup work on a old Dell running XPpro SP3 P4-3.2 w/2G RAM, so I am used to slow launches
I have 3 custom columns and 4 color rules. I leave ALL the optional windows closed.

I just slurp coffee while the Editor opens; Both Sigil and Calibre's editor take significant (15-30secs) time to open a book. Calibre stomps Sigil on overall operational running speed.
Watch your Process Explorer (a free download fro Microsoft-Sysinternals) and see where the CPU time is being spent.

Anti-Virus and Firewalls that hinder update checks (when on a network) will slow launches down.

[Adoby]

I run calibre on a linux computer with an AMD FX-8120 Eight-Core 3100 MHz processor. 16 GB fast RAM. Ubuntu 14.04. No antivirus. The computer is a little old now, from 2011, but still feels fast enough and it has very good VM performance. Cleanup library is on an Intel 530 240 GB SATA 3 SSD with /home and /. Main library on a 5400 rpm SATA 3 3TB HD. Automated backups on 2x3TB HD and a 4x4TB Synology NAS using back-in-time snapshots every hour. GeForce GTX 560 and a Philips 274ELH monitor at 1920x1080. I used to cache the spinning 3TB HDs on a 200 GB bcache on the SSD, hybrid style, but I never got it stable. Used to have /tmp in tempfs (RAM) but have it on the SSD now to conserve RAM for VMs. Considering an upgrade to 32 GB for better VM performance and to put /tmp on tempfs again. And try bcache again...

So... I have no serous issues with calibre being slow, but I avoid updating metadata in the bigger main libraries on the spinning HD, only in the smaller cleanup library on the SSD. If I attempt to update metadata in the main libraries, it is annoyingly slow. Several seconds of waiting per book...

[BetterRed]

Originally Posted by Adoby

I think the performance was significantly improved with the new database code some months(?) ago, so old posts about ways to improve performance may not be very relevant or even wrong.

I can attest to that.

Some months prior to 2013-08-23 (release date of 1.0) I had a custom column in one of my libraries that was built from a union of four other columns, when that library went past about 4K books calibre became very slow, when I removed the custom column normal service was resumed.

Sometime in September I tried reinstating the custom column, now it had no discernible effect on performance. I now have about 8K books in that library and performance remains very good.

BR

[BookJunkieLI]

So Composite Columns shouldn't cause a noticeable slowdown in Calibre? I ask because while looking for a way to clear out old Send To Device templates I found this thread - Send to device - Template help - and was giving serious thought to turning my export template into a composite column and then just pointing Send To Device to that. It currently calls 13 different columns.

[chaley]

Originally Posted by BookJunkieLI

So Composite Columns shouldn't cause a noticeable slowdown in Calibre? I ask because while looking for a way to clear out old Send To Device templates I found this thread - Send to device - Template help - and was giving serious thought to turning my export template into a composite column and then just pointing Send To Device to that. It currently calls 13 different columns.

I wouldn't go as far as to make a blanket statement "composite columns won't affect performance". They might. Why?

• If you sort on them then the must be evaluated for every book before the sort finishes.
• Same is true if you search on them. The values for every book must be computed before results can be displayed.
• Some templates require looking at the drive (mostly the formats_ functions). The result is cached but the evaluation can be slow.

Composite columns that are displayed in the spreadsheet will be evaluated as needed (as displayed). If the template is some huge program than it could be slow, but almost no one does this, and if you do then you should consider writing a custom template function.

[BookJunkieLI]

This is my export template currently.

{#library}/{#new:||/}{#fandom1:||/}{#complete2:||/}{#type:||/}{#isseries:||/}{#pairings:||/}{#fandom2:||/}{#fandom3:||/}{series:||/}{series_index:>2|| - }{title} - {author}

Not complicated at all. I'm thinking of adding

{title:shorten(9,-,5)}

but I doubt that would have much of an effect.

Out of curiosity, hiding a composite column wouldn't have any affect on it working, right? Other than to make sure it's setup right, and when I might want to make changes on it, I don't need to see the column with the export template so I was planning to hide it. That wouldn't affect the calculations or Send To Disk being able to call it, right?