MobileRead Forums - View Single Post

ZioNemo · 11-10-2017, 12:22 PM

Situation:
I have a large (~30000) book database I would like to import into Calibre.
Only a minority of those books have an actual content (~15%), the others should be added as "--empty".
I have a few additional columns (an internal book-code and some info where contents can be found).
Calbre is running as a docker container on a smart-NAS running 24/7

First attempt:
Use the remote server directly.
Failed because I wasn't able to divine some of the details of protocol (including how to send content).

Second attempt:
Use a program to repeatedly call "calibredb add", "calbredb set_metadata" and (if available) "calibredb add_format".
This somehow works, but it seems to become gradually slower and slower.
After a full 24h I had reached ~20% of the total and it was still getting slower.

First question:
Is there a practical limit to how many "books" can Calibre handle?
Does it make sense what I'm trying to do?

Third attempt:
Produce (locally) a directory structure (one book per directory) adding a "metadata.opf" file with all metadata I can get from database.
This fails because "metadata.opf" is ignored and I end up with only books having content.

Second question:
What is the advised procedure in this case?

11-10-2017, 12:22 PM	#1
ZioNemo Enthusiast Posts: 48 Karma: 10 Join Date: Mar 2008 Device: HTC-p3000	Practical limit of library size. Situation: I have a large (~30000) book database I would like to import into Calibre. Only a minority of those books have an actual content (~15%), the others should be added as "--empty". I have a few additional columns (an internal book-code and some info where contents can be found). Calbre is running as a docker container on a smart-NAS running 24/7 First attempt: Use the remote server directly. Failed because I wasn't able to divine some of the details of protocol (including how to send content). Second attempt: Use a program to repeatedly call "calibredb add", "calbredb set_metadata" and (if available) "calibredb add_format". This somehow works, but it seems to become gradually slower and slower. After a full 24h I had reached ~20% of the total and it was still getting slower. First question: Is there a practical limit to how many "books" can Calibre handle? Does it make sense what I'm trying to do? Third attempt: Produce (locally) a directory structure (one book per directory) adding a "metadata.opf" file with all metadata I can get from database. This fails because "metadata.opf" is ignored and I end up with only books having content. Second question: What is the advised procedure in this case?