Tips to speed up adding books?

[sealbeater]

So, I'm a somewhat new user of calibre and I'm currently using the command line to add books. I have quite a few and I understand that it isn't a good idea apparently to use tools like parallel to write multiple entries to the calibre db. I'm currently using a for loop that does a find on the available formats that calibre can import and after importing, deletes the file.

for I in `find . -type f -name "*.azw" -o -name "*.azw3" -o -name "*.azw4" -o -name "*.chm" -o -name "*.djvu" -o -name "*.rb" -o -name "*.snb" -o -name "*.tcr" -o -name "*.epub" -o -name "*.lit" -o -name "*.lrf" -o -name "*.mobi" -o -name "*.pdf" -o -name "*.rtf" -o -name "*.prc" -o -name "*.pdb" -o -name "*.txt" -o -name "*.doc"` ; do calibredb add --duplicates "$I" && rm -v "$I"; done


So, this works fine but I was wondering if anyone had done anything interesting to either speed up the process or multithread it somehow. I had a friend who used a hadoop cluster to speed up jobs, I was wondering if anyone had done something similar with calibre.

[kovidgoyal]

Do not call calibredb separately for each file. Every time you call calibredb, it has to load the entire db into memory. Instead use the multiple book adding options of calibredb.

It is on my todo list to add a a client/server mode for calibredb, so you can avoid this overhead.

[sealbeater]

I see. Ok, I'll give it a try, thank you very much!

[eschwartz]

Client-server model for calibredb to come alongside the client-server GUI inside the content server?

Sounds cool anyway -- looking forward to it!

[kovidgoyal]

Since the server will be getting write support, there are various things that will become possible with it eventually, such a client server mode for calibredb, directly opening libraries that reside behind a server in the gui using websockets, and so on.

However, that's a long road, so dont get too excited.

[eschwartz]

Yup -- I am aware there is a long way to go. It's still interesting to get a long-term view of the plan.