calibredb slowness: is it expected?

[ZioNemo]

Hi,
my program to automatically transport metadata to Clibre seems to work ok (kudos to Calibre developers!).
Remaining issue (side from the, apparently harmless, exception seen yesterday) is a relative slowness of calibredb.
It takes several seconds to process an entry (I am already cumulating several field updates in a single "calibredb set_metadata").
I am less 1/8 through my large lib (24000+ books) after 12 solid hours.
If this is the pace I need more than four days for a full pass!
Is this "as expected" or am I doing something wrong?

I am currently using:

1. "calibredb catalog" to get the list of known books.
2. If book is known use "calibredb catalog -i <ID>" to get metadata
3. otherwise use "calibredb add -e" to create it and get ID
4. "calibredb set_metadata <ID> -f <field>:<val> -f <field>:<val> ..." to set missing metadata (only if some metadata changed, but now I'm importing, so all books are "unknown" and thus empty).

This means I do two calibredb calls per-book (the first call is out-of-the-loop).
Is there some way to speed up operations?

TiA

[kovidgoyal]

every time you call calibredb, the entire database has to be read from disk and loaded into memory (that is the way calibre works). Therefore, calling calibredb repeatedly is never going to be very fast.

The only way to speed it up is to use the calibre db interface directly via a python script, which will avoid re-reading the db each time. But since your import is a one time operation, if I were you. I'd just let it chug away, it'll get htere eventually. Do something else in the meantime.

[Adoby]

The most obvious way to speed things up would be to use a fast computer with plenty of memory for disk caches and fast hard drives. SSD or even a RAM-drive. Especially the temp folder.

Also try to disable everything not needed, like unplug the network and disable antivirus, while the program is running.

[ZioNemo]

Yup!
It IS chugging away...
I just hope I didn't forget something essential and I'll have to redo it

Obvious solution for calibredb slowness would be to use it to "pass over" commands to a running Calibre (GUI or server), but that would mean a lot of work and I'm unsure it would be worthwhile.

I know a lot of programming languages, but Python is not among them, unfortunately...
perhaps it's time to correct this oversight

Thanks, for the time being

[ZioNemo]

Originally Posted by Adoby

The most obvious way to speed things up would be to use a fast computer with plenty of memory for disk caches and fast hard drives. SSD or even a RAM-drive. Especially the temp folder.

FYI: I moved the newly created Calibre lib to my smallish SSD and it improved dramatically performance (at least one order of magnitude).
It is now almost bearable
Good catch Adoby!

[meeera]

Originally Posted by ZioNemo

FYI: I moved the newly created Calibre lib to my smallish SSD and it improved dramatically performance (at least one order of magnitude).

Just another voice on this one - Calibre used to be a little slow on my ~5 yo laptop; not horribly so, but startup and big imports/conversions chugged away for a while. Upgraded to a 1yo laptop with SSD and now most operations happen in a few seconds or less.