Adding large number of books to Calibre got signifficantly faster. Thank you Kovid

[kacir]

Kovid,
you are master of the understatement.
On the changelog page you write: "Add Books: Improve performance when adding large numbers of books at once"
This is HUGE improvement. The books are, at the moment, being added more than order of magnitude faster. On my system I see more than 30 fold improvement in speed.

Recently I have purchased an SSD disk, hoping that, among other things, adding books would be a little bit faster. There are many things that are much faster, but until recently I did not see any improvement in the speed of book adding.
An hour ago I have upgraded to 2.10 and I was blown away with the speed increase.

THANK YOU!

[kovidgoyal]

You're welcome

The improvement comes mainly from changing the algorithm to detect duplicates from an O(n) to an O(1) implementation (using a hash table). And making the detection use multiple processor cores.

[HarryT]

Kovid, I've just experienced a problem which may be related to this.

I've done my usual weekly task of running "Kindle for PC", downloading the books I've bought in the last week (14 in total), and adding them to Calibre. When I did this, a ... certain plugin, which is a "File Type" plugin ... apparently randomly failed to work for some of the books that were added; they stayed as "AZW" files, rather than being converted to MOBI or AZW3, as is normally the case. I deleted the books, and re-added them. Same thing, but different books were successfully processed. I deleted the books again, and this time added them one by one: this time they were all correctly processed. Is it possible that the optimisation process you've added means that books are no longer being processed by file-type plugins?

[kovidgoyal]

As far as I know file type plugins should continue to work normally. There were no changes to that part of the code. In fact the code to add them one by one is exactly the same as the code to add them in large numbers, its just that for single adds the large number is one. However, since I have no idea what that plugin does internally, I cant be sure.

Run calibre in debug mode and see if anything is printed out when the problem happens.

[HarryT]

Thanks for the suggestion. I'll do it when I add books next week, and let you know. Can you remind me how to run Calibre in debug mode? Run it from the command line and add a "-debug" parameter?

[kovidgoyal]

Right click the preferences button

[HarryT]

Thanks!

[kacir]

Originally Posted by HarryT

Can you remind me how to run Calibre in debug mode? Run it from the command line and add a "-debug" parameter?

I personally use command
calibre-debug -g
but I use Linux and I like to use console when looking into problems.
You need the -g switch to start it in graphical user mode.

[HarryT]

Tried the debug. The plugin in question generally prints a "Processing..." and a "Successfully processed..." message for each book, but when presented with a batch of books is now (for whatever reason) not getting as far as the second message - it's just silently failing. Not a Calibre problem - I'll ask elsewhere.

[eschwartz]

Originally Posted by kacir

I personally use command
calibre-debug -g
but I use Linux and I like to use console when looking into problems.
You need the -g switch to start it in graphical user mode.

I added the following script as

~/bin/calibre

#!/bin/bash

if [ $# -eq 0 ] || [[ "$1" = "--detach" ]]; then
    echo -e "\n\n\nStarting calibre at $(date).\n\n\n" >> /tmp/calibre-debug-$USER.log 2>&1
    calibre-debug -g 2>&1 | tee -a /tmp/calibre-debug-$USER.log 
else
    /usr/bin/calibre $@
fi

Then I just run via "calibre", or use the calibre.desktop launcher.

This way, I am ALWAYS in debug mode, just in case.