Library Size

[Tanjamuse]

I just remembered Chaley's answer in this thread:

https://www.mobileread.com/forums/sho...39&postcount=8

I was wondering, doesn't that mean that my custom columns are stored in the metadata.db?

[eschwartz]

I see no indication there one way or another.

Custom columns haven't changed -- insomuch as they have always bee stored in the database.
custom columns built from other columns -- those store the template once for the entire column, then calculate the value for each row dynamically.
Performing functions on all the metadata columns in order to get the value of a custom column built from other columns is slow.

If you use custom columns built from other columns, then calibre will be slower -- no matter what. Sometimes a lot slower -- it depends how many you have.

[Tanjamuse]

Would it help if I removed them from the library I download to?

[DaltonST]

@Tanjamuse:

Not a half-million, but sufficiently large enough to give me a lot of personal experience that I can share with you in the form of suggestions derived from what I have had to do:

(1) Create a new empty "pattern" Calibre Library that has all of the custom columns and plugin configurations that you can foresee needing or wanting.

(2) Create new Calibre Libraries using your "pattern" metadata.db and its associated .json.

(3) Create one new Calibre Library for each forseeable high-level "Genre". For example, as a minimum, 3: Fiction; Factual; Poetry.

(4) Based on your anticipated book-count by high-level "Genre", create more detailed new Calibre Libraries using your "pattern" metadata.db and its associated .json.

(5) Move your books in your original 9,000 book library into their new homes, and then eliminate duplicates.

(6) Count what remains in each Library, and create new "pattern" Libraries as needed to further break down the "Genre" represented in a particular Library if it is too voluminous. Example: You may need to create separate Libraries for "Factual:History", "Factual:Autobiographies&Biographies" etc.

(7) Create new Virtual Libraries based on the final metadata in each of your pattern Libraries. Example: VLs of "Factual:Autobiographies" and "Factual:Biographies" for the "Factual:Autobiographies&Biographies Library.

(8) You will want to compress the metadata.db files frequently. I suggest using .bat files that invoke the calibredb command-line utility to "check_library", which not only tells you of any corruption, but also vacuums/compresses/defragments a Library. You can create a single .bat file that sequentially executes calibredb for each of your libraries in turn. See: http://manual.calibre-ebook.com/cli/calibredb.html

Personally, I would not put all of the libraries on a single physical hard-drive unless it was a reliable RAID device. Too risky. Backing up 500,000 metadata.opf files plus 500,000 cover.jpg files, which are "small" files, takes a very long time. Obviously you need to back up everything initially. I back up all of my metadata.db files daily to multiple external devices, since "small" files take much too long to do, and Calibre generates new metadata.opf files automatically regardless of whether you want it to or not.

Good luck.



DaltonST

[Tanjamuse]

Would it help if it's 250.000 books and no custom columns build from other columns?