Practical limit of library size.

[ZioNemo]

Situation:
I have a large (~30000) book database I would like to import into Calibre.
Only a minority of those books have an actual content (~15%), the others should be added as "--empty".
I have a few additional columns (an internal book-code and some info where contents can be found).
Calbre is running as a docker container on a smart-NAS running 24/7

First attempt:
Use the remote server directly.
Failed because I wasn't able to divine some of the details of protocol (including how to send content).

Second attempt:
Use a program to repeatedly call "calibredb add", "calbredb set_metadata" and (if available) "calibredb add_format".
This somehow works, but it seems to become gradually slower and slower.
After a full 24h I had reached ~20% of the total and it was still getting slower.

First question:
Is there a practical limit to how many "books" can Calibre handle?
Does it make sense what I'm trying to do?

Third attempt:
Produce (locally) a directory structure (one book per directory) adding a "metadata.opf" file with all metadata I can get from database.
This fails because "metadata.opf" is ignored and I end up with only books having content.

Second question:
What is the advised procedure in this case?

[kovidgoyal]

You absolutely must not call calibredb repeatedly, use calibredb with a remote server instead. Otherwise every time you call it it has to laod the entire datbase. And many people manage libraries of 100K books in calibre.

[BetterRed]

@ZioNemo - if you can massage your 'database' into a CSV you can probably make use of the Import List optional plugin -- especially for the empty books.

This wizard-based plugin allows you to match existing or create empty books in calibre based on lists of books from external sources.

BR

[ZioNemo]

Sorry Kovid, I fail to understand You.

I *am* using calibredb with a remote server ("calibredb <command> --library-path=http://syno0:38081/#lib --username=<mself> --password=<secret>")

... but I'm also calling it multiple times: for each book I do:
1) calibredb search (to get id if book exists)
2) calibredb add --empty --title=<Title> --authors=<Authors> --isbn=<ISBN> (if book did not exist)
3) calibredb set_metadata <id> --field=identifiers:isbn:<ISBN>,dewey:<Dewey> --field=#collecton --field=publisher:<publisher> ...
4) calibredb set_metadata <id> --field=cover:<coverImage>
5) calibredb add_format <id> --don't-replace <content file>

I could possibly collapse (3 and 4) and (2 and 5) but it still leaves me with 3 calibredb calls for each book.

As said I also tried preparing a local directory structure (one book per directory) and use "calibredb -1 -r", but that completely ignores the "metadata.opf" file and, since most of content files do not have metadata, even title is mostly bogus.

What should I do, exactly?

[ZioNemo]

Thanks @BetterRed.

I used the "Import List" plugin and it seems what I need.
Only problem is I didn't find a way to add content (aka: "format") to the books.
Did I miss something?

Thanks.

[theducks]

Originally Posted by ZioNemo

Thanks @BetterRed.

I used the "Import List" plugin and it seems what I need.
Only problem is I didn't find a way to add content (aka: "format") to the books.
Did I miss something?

Thanks.

formats: is a Calibre Computed field that is based upon what it formats it physically adds.
Import List can create an 'empty' format

If you have a custom column, like I do, for a 'Original Format'. Then somehow the PI would need to be able to resolve that fields name (and use it).

[BetterRed]

Originally Posted by ZioNemo

Thanks @BetterRed.

I used the "Import List" plugin and it seems what I need.
Only problem is I didn't find a way to add content (aka: "format") to the books.
Did I miss something?

Thanks.

@ZioNemo - Import List only deals with metadata, hence my "especially for the empty books"

You could add the 'format' files to the empty book records with the calibredb add_format command, but you need to specify the book number.

Or you could use the 'one book per directory' Add method with an opf file in each - Third attempt in your original post.

BR

[ZioNemo]

Originally Posted by BetterRed

@ZioNemo - Import List only deals with metadata, hence my "especially for the empty books"

You could add the 'format' files to the empty book records with the calibredb add_format command, but you need to specify the book number.

Or you could use the 'one book per directory' Add method with an opf file in each - Third attempt in your original post.

BR

I still have a couple of problems:

• I went for real and Import List was rather quick parsing the .csv file and to do the preview, but after I gave the final "Finish" started trashing on disk and it's still mumbling with no feedback aside for disk access (It's a couple of hours, now)
• I don't really understand your comment about using my "third attempt" with the .opf file: I tried that, but Calibre seems to ignore completely the .opf and tries to gleam Title/author from unknown (to me) sources with "funny results". What should I do to prepare a directory that will be associated to a given book (note: I have some books "duplicate" because they are different editions.

TiA!

[BetterRed]

Originally Posted by ZioNemo

I still have a couple of problems:

• I went for real and Import List was rather quick parsing the .csv file and to do the preview, but after I gave the final "Finish" started trashing on disk and it's still mumbling with no feedback aside for disk access (It's a couple of hours, now)

It's better to post questions about optional plugins, such as Import List, in the the corresponding thread in the Plugins subforum.

But regarding the speed of processing the csv; when you use IL to add empty books, it will write a subfolder for every primary author, a subfolder for every book, and write an opf file for every book. By my reckoning, for ~25.000 'empty' books, that's about 70,000 operations, plus a similar (or probably more) database operations.

Calibre is optimised to provide robust file operations on file systems ranging from FAT32 to Ext4 and most others in between.

Speed is not Calibre's forte on file system intensive tasks - whereas searching in a library is.

Originally Posted by ZioNemo

• I don't really understand your comment about using my "third attempt" with the .opf file: I tried that, but Calibre seems to ignore completely the .opf and tries to gleam Title/author from unknown (to me) sources with "funny results". What should I do to prepare a directory that will be associated to a given book (note: I have some books "duplicate" because they are different editions.

If you have a folder/file structure, such as that shown below and the metadata.opf files comply with calibre's expectations regarding their contents and formation then using the - Add from directories [ . . . ] (One book per directory [ . . . ] should pick up the metadata from the opf files.

Calibre's .opf files are not normally created by anything other than calibre itself. The primary reason they can be used in this way is that they provide a means of reconstructing a library database from an pre-existing calibre library.

Speaking for myself, I wouldn't dream of creating the metadata.opf files outside of calibre, far. far too fiddlesome. I'd get the book's (format files) into a library, get the Author names and Book Titles correct, deal with duplicates, and then use IL to match on Author and Title to inject other metadata columns (publisher, pubdate, series etc)

    Title 1
        metadata.opf
        book1.epub 
    Title 2
        metadata.opf
        book3.epub 
    Title 3
        metadata.opf
        book3.epub 
        book3.pdf
    Title 4
        metadata.opf
        book4.pdf

TiA!

Hope this helps

BR