Register Guidelines E-Books Today's Posts Search

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 11-10-2017, 12:22 PM   #1
ZioNemo
Enthusiast
ZioNemo began at the beginning.
 
Posts: 48
Karma: 10
Join Date: Mar 2008
Device: HTC-p3000
Practical limit of library size.

Situation:
I have a large (~30000) book database I would like to import into Calibre.
Only a minority of those books have an actual content (~15%), the others should be added as "--empty".
I have a few additional columns (an internal book-code and some info where contents can be found).
Calbre is running as a docker container on a smart-NAS running 24/7

First attempt:
Use the remote server directly.
Failed because I wasn't able to divine some of the details of protocol (including how to send content).

Second attempt:
Use a program to repeatedly call "calibredb add", "calbredb set_metadata" and (if available) "calibredb add_format".
This somehow works, but it seems to become gradually slower and slower.
After a full 24h I had reached ~20% of the total and it was still getting slower.

First question:
Is there a practical limit to how many "books" can Calibre handle?
Does it make sense what I'm trying to do?

Third attempt:
Produce (locally) a directory structure (one book per directory) adding a "metadata.opf" file with all metadata I can get from database.
This fails because "metadata.opf" is ignored and I end up with only books having content.

Second question:
What is the advised procedure in this case?
ZioNemo is offline   Reply With Quote
Old 11-10-2017, 12:58 PM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 45,345
Karma: 27182818
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
You absolutely must not call calibredb repeatedly, use calibredb with a remote server instead. Otherwise every time you call it it has to laod the entire datbase. And many people manage libraries of 100K books in calibre.
kovidgoyal is online now   Reply With Quote
Advert
Old 11-10-2017, 02:29 PM   #3
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 21,718
Karma: 29711016
Join Date: Mar 2012
Location: Sydney Australia
Device: none
@ZioNemo - if you can massage your 'database' into a CSV you can probably make use of the Import List optional plugin -- especially for the empty books.

Quote:
This wizard-based plugin allows you to match existing or create empty books in calibre based on lists of books from external sources.
BR
BetterRed is offline   Reply With Quote
Old 11-10-2017, 02:48 PM   #4
ZioNemo
Enthusiast
ZioNemo began at the beginning.
 
Posts: 48
Karma: 10
Join Date: Mar 2008
Device: HTC-p3000
Sorry Kovid, I fail to understand You.

I *am* using calibredb with a remote server ("calibredb <command> --library-path=http://syno0:38081/#lib --username=<mself> --password=<secret>")

... but I'm also calling it multiple times: for each book I do:
1) calibredb search (to get id if book exists)
2) calibredb add --empty --title=<Title> --authors=<Authors> --isbn=<ISBN> (if book did not exist)
3) calibredb set_metadata <id> --field=identifiers:isbn:<ISBN>,dewey:<Dewey> --field=#collecton --field=publisher:<publisher> ...
4) calibredb set_metadata <id> --field=cover:<coverImage>
5) calibredb add_format <id> --don't-replace <content file>

I could possibly collapse (3 and 4) and (2 and 5) but it still leaves me with 3 calibredb calls for each book.

As said I also tried preparing a local directory structure (one book per directory) and use "calibredb -1 -r", but that completely ignores the "metadata.opf" file and, since most of content files do not have metadata, even title is mostly bogus.

What should I do, exactly?
ZioNemo is offline   Reply With Quote
Old 11-10-2017, 05:42 PM   #5
ZioNemo
Enthusiast
ZioNemo began at the beginning.
 
Posts: 48
Karma: 10
Join Date: Mar 2008
Device: HTC-p3000
Thanks @BetterRed.

I used the "Import List" plugin and it seems what I need.
Only problem is I didn't find a way to add content (aka: "format") to the books.
Did I miss something?

Thanks.
ZioNemo is offline   Reply With Quote
Advert
Old 11-10-2017, 06:42 PM   #6
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 31,047
Karma: 60358908
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
Quote:
Originally Posted by ZioNemo View Post
Thanks @BetterRed.

I used the "Import List" plugin and it seems what I need.
Only problem is I didn't find a way to add content (aka: "format") to the books.
Did I miss something?

Thanks.
formats: is a Calibre Computed field that is based upon what it formats it physically adds.
Import List can create an 'empty' format

If you have a custom column, like I do, for a 'Original Format'. Then somehow the PI would need to be able to resolve that fields name (and use it).
theducks is offline   Reply With Quote
Old 11-10-2017, 08:40 PM   #7
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 21,718
Karma: 29711016
Join Date: Mar 2012
Location: Sydney Australia
Device: none
Quote:
Originally Posted by ZioNemo View Post
Thanks @BetterRed.

I used the "Import List" plugin and it seems what I need.
Only problem is I didn't find a way to add content (aka: "format") to the books.
Did I miss something?

Thanks.
@ZioNemo - Import List only deals with metadata, hence my "especially for the empty books"

You could add the 'format' files to the empty book records with the calibredb add_format command, but you need to specify the book number.

Or you could use the 'one book per directory' Add method with an opf file in each - Third attempt in your original post.

BR
BetterRed is offline   Reply With Quote
Old 11-11-2017, 08:53 AM   #8
ZioNemo
Enthusiast
ZioNemo began at the beginning.
 
Posts: 48
Karma: 10
Join Date: Mar 2008
Device: HTC-p3000
Quote:
Originally Posted by BetterRed View Post
@ZioNemo - Import List only deals with metadata, hence my "especially for the empty books"

You could add the 'format' files to the empty book records with the calibredb add_format command, but you need to specify the book number.

Or you could use the 'one book per directory' Add method with an opf file in each - Third attempt in your original post.

BR
I still have a couple of problems:
  • I went for real and Import List was rather quick parsing the .csv file and to do the preview, but after I gave the final "Finish" started trashing on disk and it's still mumbling with no feedback aside for disk access (It's a couple of hours, now)
  • I don't really understand your comment about using my "third attempt" with the .opf file: I tried that, but Calibre seems to ignore completely the .opf and tries to gleam Title/author from unknown (to me) sources with "funny results". What should I do to prepare a directory that will be associated to a given book (note: I have some books "duplicate" because they are different editions.
TiA!
ZioNemo is offline   Reply With Quote
Old 11-11-2017, 03:52 PM   #9
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 21,718
Karma: 29711016
Join Date: Mar 2012
Location: Sydney Australia
Device: none
Quote:
Originally Posted by ZioNemo View Post
I still have a couple of problems:
  • I went for real and Import List was rather quick parsing the .csv file and to do the preview, but after I gave the final "Finish" started trashing on disk and it's still mumbling with no feedback aside for disk access (It's a couple of hours, now)
It's better to post questions about optional plugins, such as Import List, in the the corresponding thread in the Plugins subforum.

But regarding the speed of processing the csv; when you use IL to add empty books, it will write a subfolder for every primary author, a subfolder for every book, and write an opf file for every book. By my reckoning, for ~25.000 'empty' books, that's about 70,000 operations, plus a similar (or probably more) database operations.

Calibre is optimised to provide robust file operations on file systems ranging from FAT32 to Ext4 and most others in between.

Speed is not Calibre's forte on file system intensive tasks - whereas searching in a library is.
Quote:
Originally Posted by ZioNemo View Post
  • I don't really understand your comment about using my "third attempt" with the .opf file: I tried that, but Calibre seems to ignore completely the .opf and tries to gleam Title/author from unknown (to me) sources with "funny results". What should I do to prepare a directory that will be associated to a given book (note: I have some books "duplicate" because they are different editions.
If you have a folder/file structure, such as that shown below and the metadata.opf files comply with calibre's expectations regarding their contents and formation then using the - Add from directories [ . . . ] (One book per directory [ . . . ] should pick up the metadata from the opf files.

Calibre's .opf files are not normally created by anything other than calibre itself. The primary reason they can be used in this way is that they provide a means of reconstructing a library database from an pre-existing calibre library.

Speaking for myself, I wouldn't dream of creating the metadata.opf files outside of calibre, far. far too fiddlesome. I'd get the book's (format files) into a library, get the Author names and Book Titles correct, deal with duplicates, and then use IL to match on Author and Title to inject other metadata columns (publisher, pubdate, series etc)
Code:
    Title 1
        metadata.opf
        book1.epub 
    Title 2
        metadata.opf
        book3.epub 
    Title 3
        metadata.opf
        book3.epub 
        book3.pdf
    Title 4
        metadata.opf
        book4.pdf


TiA!
Hope this helps

BR

Last edited by BetterRed; 11-11-2017 at 04:00 PM. Reason: clarity
BetterRed is offline   Reply With Quote
Reply

Tags
add books, add empty books, remote library


Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
library size limit? big pdf files mkeINjp Library Management 6 03-31-2016 09:34 AM
Limit to font size? dandelioncottage Marvin 0 11-08-2014 12:03 PM
Practical size limit on epubs bobcdy ePub 4 11-13-2010 04:13 AM
Size limit on PDFs? tench Sony Reader 4 06-24-2009 06:48 PM
PDF size limit sic Sony Reader 14 07-15-2007 04:50 AM


All times are GMT -4. The time now is 12:35 PM.


MobileRead.com is a privately owned, operated and funded community.