Slow calibredb add on huge ebooks directory

[ternyk]

Hi,
I'm trying to import huge ebook collection (30GB) by a command line using calibredb.
The command I'm using is:
find . -type f| while read filename; do calibredb add -d -r --isbn "$filename%.*" "$filename"; done;
It's ok but veeery slow! After adding each of the files there is an output message:

0123456789 not found [my note: it's the isbn of the book]
Backing up metadata
Notifying calibre of the change

It seems that calibre is doing too much I want. I don't need backing up the database. Only just fast copying with adding the info to the databse. Is there any option to disable some internal processing when adding to the library?

(I've tried from GUI but it finished with error saying that the operation was aborted as it took too long - it was about 1 hour.)

[itimpi]

I do not think that message relates to backing up the database. I think it just means that after adding the book, the metadata.opf file has been created for that book (although I guess I could be wrong). Adding books is always a slow(ish) process as Calibre copies each file into its library as well as setting up entries in the metadata.db database. If you are reading metadata from the file it also has to open each file to extract its metadata.

One way that can speed up adding books (particularily for PDF files) is to disable reading the metadata from the file. However this may not be desireable as it means you later have to look at each book if you want to get its metadata set up.

[ternyk]

Thanks. I see.
I think that I removed reading metadata last time I was using GUI. Can I disable reading metadata from the file from the command line parameter? I don't have access to the GUI now, just using ssh.

[Manichean]

I don't think you can, the manual notes no such option. You could try disabling the relevant metadata reader plugins in the GUI, but I don't know if that carries over to the CLI.

[kovidgoyal]

Originally Posted by Manichean

I don't think you can, the manual notes no such option. You could try disabling the relevant metadata reader plugins in the GUI, but I don't know if that carries over to the CLI.

It does. And you can disable plugins from the CLI using calibre-customize

[ternyk]

Originally Posted by kovidgoyal

It does. And you can disable plugins from the CLI using calibre-customize

Thanks, I've missed it.

I had the option for reading the metadata disabled.

Finally, importing has completed. Uff, it took nearly 8 hours to add 5000 documents.

But I forgot to mention, that it was working on the ntfs file system under Linux so it also had an impact on the overall process.

[Manichean]

Originally Posted by kovidgoyal

It does. And you can disable plugins from the CLI using calibre-customize

Good to know the CLI respects the plugin settings.

[ternyk]

Last question: how can I download and update metadata & covers for the selected/all books but from command line?
It seems that I should combine: fetch-ebook-metadata with ebook-meta but maybe is there a simpler way?

[ternyk]

OK. I've done it. In case someone will be looking for something similiar I'm putting the script.


#!/bin/sh

mkdir opf_tmp

while read id isbn ; do
echo "$id $isbn"
fetch-ebook-metadata -o OPF -i "$isbn" > ./opf_tmp/"$isbn"
calibredb set_metadata "$id" "./opf_tmp/$isbn"
done < <(calibredb list)