Slow calibredb add on huge ebooks directory

ternyk · 05-16-2011, 07:30 AM

Hi,
I'm trying to import huge ebook collection (30GB) by a command line using calibredb.
The command I'm using is:
find . -type f| while read filename; do calibredb add -d -r --isbn "$filename%.*" "$filename"; done;
It's ok but veeery slow! After adding each of the files there is an output message:

0123456789 not found [my note: it's the isbn of the book]
Backing up metadata
Notifying calibre of the change

It seems that calibre is doing too much I want. I don't need backing up the database. Only just fast copying with adding the info to the databse. Is there any option to disable some internal processing when adding to the library?

(I've tried from GUI but it finished with error saying that the operation was aborted as it took too long - it was about 1 hour.)

itimpi · 05-16-2011, 07:43 AM

I do not think that message relates to backing up the database. I think it just means that after adding the book, the metadata.opf file has been created for that book (although I guess I could be wrong). Adding books is always a slow(ish) process as Calibre copies each file into its library as well as setting up entries in the metadata.db database. If you are reading metadata from the file it also has to open each file to extract its metadata.

One way that can speed up adding books (particularily for PDF files) is to disable reading the metadata from the file. However this may not be desireable as it means you later have to look at each book if you want to get its metadata set up.

ternyk · 05-16-2011, 08:06 AM

Thanks. I see.
I think that I removed reading metadata last time I was using GUI. Can I disable reading metadata from the file from the command line parameter? I don't have access to the GUI now, just using ssh.

Manichean · 05-16-2011, 10:00 AM

I don't think you can, the manual notes no such option. You could try disabling the relevant metadata reader plugins in the GUI, but I don't know if that carries over to the CLI.

kovidgoyal · 05-16-2011, 10:53 AM

Quote:

Originally Posted by Manichean

I don't think you can, the manual notes no such option. You could try disabling the relevant metadata reader plugins in the GUI, but I don't know if that carries over to the CLI.

It does. And you can disable plugins from the CLI using calibre-customize

ternyk · 05-16-2011, 11:07 AM

Quote:

Originally Posted by kovidgoyal

It does. And you can disable plugins from the CLI using calibre-customize

Thanks, I've missed it.

I had the option for reading the metadata disabled.

Finally, importing has completed. Uff, it took nearly 8 hours to add 5000 documents.

But I forgot to mention, that it was working on the ntfs file system under Linux so it also had an impact on the overall process.

Manichean · 05-16-2011, 11:09 AM

Quote:

Originally Posted by kovidgoyal

It does. And you can disable plugins from the CLI using calibre-customize

Good to know the CLI respects the plugin settings.

ternyk · 05-17-2011, 02:52 AM

Last question: how can I download and update metadata & covers for the selected/all books but from command line?
It seems that I should combine: fetch-ebook-metadata with ebook-meta but maybe is there a simpler way?

ternyk · 05-17-2011, 04:39 AM

OK. I've done it. In case someone will be looking for something similiar I'm putting the script.

#!/bin/sh

mkdir opf_tmp

while read id isbn ; do
echo "$id $isbn"
fetch-ebook-metadata -o OPF -i "$isbn" > ./opf_tmp/"$isbn"
calibredb set_metadata "$id" "./opf_tmp/$isbn"
done < <(calibredb list)

05-16-2011, 07:30 AM	#1
ternyk Member Posts: 10 Karma: 10 Join Date: Feb 2011 Device: Kindle 3	Slow calibredb add on huge ebooks directory Hi, I'm trying to import huge ebook collection (30GB) by a command line using calibredb. The command I'm using is: find . -type f\| while read filename; do calibredb add -d -r --isbn "$filename%." "$filename"; done; It's ok but veeery slow! After adding each of the files there is an output message: 0123456789 not found [my note: it's the isbn of the book] Backing up metadata Notifying calibre of the change It seems that calibre is doing too much I want. I don't need backing up the database. Only just fast copying with adding the info to the databse. Is there any option to disable some internal processing when adding to the library? (I've tried from GUI but it finished with error saying that the operation was aborted as it took too long - it was about 1 hour.) Last edited by ternyk; 05-16-2011 at 07:36 AM. Reason: typo in the command line*

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Metadata during directory add?	jdege	Library Management	6	05-15-2011 01:02 PM
Calibredb add after ebook-convert fails	spullar	Calibre	5	03-04-2010 01:14 PM
I Can't add a directory	takelu	Calibre	2	02-05-2010 07:21 PM
Should Calibre take a long time to add a directory full of ebooks?	jusmee	Calibre	4	10-25-2009 10:34 PM
What format to use for huge HTML directory -> iPhone?	Robotech_Master	Workshop	6	12-22-2008 04:59 PM

05-16-2011, 07:43 AM	#2
itimpi Wizard Posts: 4,553 Karma: 950151 Join Date: Nov 2008 Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)	I do not think that message relates to backing up the database. I think it just means that after adding the book, the metadata.opf file has been created for that book (although I guess I could be wrong). Adding books is always a slow(ish) process as Calibre copies each file into its library as well as setting up entries in the metadata.db database. If you are reading metadata from the file it also has to open each file to extract its metadata. One way that can speed up adding books (particularily for PDF files) is to disable reading the metadata from the file. However this may not be desireable as it means you later have to look at each book if you want to get its metadata set up.

05-16-2011, 08:06 AM	#3
ternyk Member Posts: 10 Karma: 10 Join Date: Feb 2011 Device: Kindle 3	Thanks. I see. I think that I removed reading metadata last time I was using GUI. Can I disable reading metadata from the file from the command line parameter? I don't have access to the GUI now, just using ssh.

05-16-2011, 10:00 AM	#4
Manichean Wizard Posts: 3,130 Karma: 91256 Join Date: Feb 2008 Location: Germany Device: Cybook Gen3	I don't think you can, the manual notes no such option. You could try disabling the relevant metadata reader plugins in the GUI, but I don't know if that carries over to the CLI.

05-17-2011, 02:52 AM	#8
ternyk Member Posts: 10 Karma: 10 Join Date: Feb 2011 Device: Kindle 3	Last question: how can I download and update metadata & covers for the selected/all books but from command line? It seems that I should combine: fetch-ebook-metadata with ebook-meta but maybe is there a simpler way?

05-17-2011, 04:39 AM	#9
ternyk Member Posts: 10 Karma: 10 Join Date: Feb 2011 Device: Kindle 3	OK. I've done it. In case someone will be looking for something similiar I'm putting the script. #!/bin/sh mkdir opf_tmp while read id isbn ; do echo "$id $isbn" fetch-ebook-metadata -o OPF -i "$isbn" > ./opf_tmp/"$isbn" calibredb set_metadata "$id" "./opf_tmp/$isbn" done < <(calibredb list)