[GUI Plugin] Consolidate All Library Metadata (CALM)

[DaltonST]

[GUI Plugin] Consolidate All Library Metadata (CALM): A Calibre Add-on

Application Functions and Features:

Spoiler:

[1] Consolidate a snapshot of all of the standard and custom column metadata for all of your Calibre Libraries into a temporary CALM Target Library.

[2] View the consolidated metadata of all of your books from all of your Calibre Libraries simultaneously;

[3] Standardize Tags, Genres, Identifiers and most Custom Columns centrally in the snapshot CALM Target Library using CALM's Metadata Tools;

[4] Propagate the standardized Tags, Genres, Identifiers and most Custom Columns from the snapshot CALM Target Library back to the 'real' Source Libraries;

Questions & Answers:

Spoiler:

[1] Do I need CALM if I have only a single Calibre Library?

The answer is often 'Yes'. CALM has Metadata Tools for standardizing and deriving Tags and Genres using Rules that you maintain based on your personal needs and desires.

If you do not use Tags and Genres, or if they are already standardized, and if you do not frequently add new books to your single Calibre Library, then you do not need CALM.


[2] I have many Libraries, but my metadata is already 'perfect' and entirely standardized. What does CALM have to offer me?

Consolidation of the basic metadata for all of the books in all of your libraries so you can view and analyze it all simultaneously using both Calibre advanced search functions and the MultiColumnSearch plug-in's additional advanced search capabilities.

Afterward, using CALM-specific functionality, you can shuffle your books among your many libraries for the purpose of moving them into their correct locations, as well as deleting cross-library duplicates.


[3] Is CALM's Consolidation Job just a mass “copy” from all of my Calibre Libraries into the CALM Target Library?

No. Absolutely not. The Consolidation Job creates a temporary “snapshot” Calibre Library that has no book formats whatsoever. It copies the standard metadata for each book in each specified “Source Library”, but not the book itself. For that reason, the consolidation job executes quickly.


[4] Does CALM also consolidate custom columns?

Yes. All, some or none. You decide.


[5] Is the CALM Target Library a fully functional, "real", Calibre Library?

Yes. Of course. Otherwise, you would not be able to use any Calibre plugins, including the ones recommended below.

Requires Minimum Calibre Version: 6.0.0


Companion Plugin - Essential : View Manager


Companion Plugin - Recommended : CalibreSpy


Companion Plugin - Recommended : Job Spy


Companion Plugin - Recommended : MultiColumnSearch


Version History:

Spoiler:

Version 2.0.46 -2023-02-05 Case-sensitive filesystems, like BTRFS, are now supported. Qt6 compatibility when resizing columns via the mouse cursor in the list editors for selectively activating custom columns and tag rules.

Version 2.0.45 -2023-01-26 Qt6 compatibility. Improve DEBUG for enumeration harmonization.
Version 2.0.44 -2023-01-13 Qt.core.
Version 2.0.43 -2022-07-24 Qt6 Compatibility.
Version 2.0.42 -2022-04-13 Qt6 Compatibility; Minimum Calibre Version now 6.0.0 (5.99.8+)
Version 2.0.40 -2020-12-19 "CalibreSpy Plugin" template settings for the CALM Target Library; Information and Error Messages improved; Self-repair of user-created CALM Target Library (only CALM may create the Target Library).
Version 2.0.39 -2020-12-18 Added helpful debugging information.
Version 2.0.38 -2020-12-16 Miscellaneous technical tweaks. Template Target CALM Library updated to version 24 by Calibre automatically.
Version 2.0.37 -2020-08-06 Qt tweaks for small displays
Version 2.0.36 -2020-06-26 Regression Error caused by Version 2.0.35 synching of Derive Genres causing 'CALM Icon in ToolBar' for Portable/32bit Windows bug: fixed.
Version 2.0.35 -2020-02-19 CALM 'Derive Genres' technically upgraded to be consistent with QuarantineAndScrub's 'Derive Genres'.
Version 2.0.34 -2019-12-22 Technical changes after Python 3.8 testing with Calibre 4.99.2. Minimum version now 4.6
Version 2.0.33 -2019-07-08 Python 3 compatibility; Minimum version now 3.41.3.
Version 2.0.32 -2017-06-11 Calibre Standard Schema Version of template CALM Target Library metadata.db updated to Version 23 automatically by Calibre itself.
Version 2.0.31 -2017-05-21 Technical changes.
Version 2.0.30 -2017-05-14 Technical and functional changes for Calibre 3.0.0 compatibility. Calibre Release 3.0.0 deprecated the use of calibredb while the GUI is also running. The "Library Tools" Tab has been deleted, and is no longer available for use. You will no longer be able to shuffle your books among your Calibre Library ecosystem from a single centralized control panel (i.e., the CALM Target Library). You will need to manually add, copy, and/or delete your books by switching to each different Source Library in your ecosystem. You also will no longer be able to refresh the metadata.opf files for all of your books in all of your Source Libraries with a single click of a button.
Version 2.0.29 -2017-05-05 Allow themes with user-defined icons.
Version 2.0.28 -2016-08-24 'Purge_all_enumerations' message fixed.
Version 2.0.27 -2016-08-22 Enumerations changed.
Version 2.0.26 -2016-08-12 Calibre Release 2.64 implemented new attributes for Custom Column datatype=comments. CALM was changed to be aware of the new attributes during Generation.
Version 2.0.25 -2016-07-30 Bug fix for OSX and Linux.
Version 2.0.24 -2016-07-30 Bug fix in library path sorting for OSX and Linux.
Version 2.0.23 -2016-07-29 Bug fix related to saving preferences within the Source Library tab if the user did not immediately move to the other tabs and execute a function that would also save, albeit indirectly, the Source Library preferences.
Version 2.0.22 -2016-07-28 Fixed Source Libraries Tab Load-All-Known-Libraries function.
Version 2.0.21 -2016-07-28 Bug Fix.
Version 2.0.20 -2016-07-28 Bug Fix.
Version 2.0.19 -2016-06-24 Compatibility improvement for use with Linux and OSX; now does not delete any Identifiers that start with a 'z', nor 'doi'.
Version 2.0.18 -2016-05-25 Technical changes.
Version 2.0.17 -2016-03-26 Attempt to solve OSX subprocess issue.
Version 2.0.16 -2016-03-05 Speed Optimization for the Consolidation of the MCS 'Word-Book Index'; Miscellaneous enhancements.
Version 2.0.15 -2016-03-04 New: Option to consolidate the 'Multi-Column Search' plug-in 'Word-Book Index' into the CALM Target Library so that MCS Word-Book Queries may be executed against all consolidated Source Libraries simultaneusly.
Version 2.0.14 -2016-02-16 Technical tweaks.
Version 2.0.13 -2016-02-03 New Metadata Tool: Copy almost any Target Library Custom Column back to the mapped Source Library Custom Column, replacing the Source with the Target. Analagous to what is already available for Tags and Genre.
Version 2.0.12 - 2016-01-12 Miscellaneous enhancements.
Version 2.0.11 - 2015-12-27 Miscellaneous enhancements.
Version 2.0.10 - 2015-12-26 Composite Custom Columns that would cause Template Key Errors in the Target Library are made inert after Generation. The cause of these errors is usually a lack of Custom Column Standardization within the Source Libraries.
Version 2.0.9 - 2015-12-24 Miscellaneous tweaks.
Version 2.0.8 - 2015-12-24 New 'Standard' CALM Target Library Custom Column: #source_cover. The CALM Template Target Library used to 'refresh' has been updated to always show that new Standard Custom Column. Unless you start with a clean slate, you will have to update View Manager yourself.
Version 2.0.7 - 2015-12-23 New 'Standard' CALM Target Library Custom Column: #source_format_types. The CALM Template Target Library used to 'refresh' has been updated to always show that new Standard Custom Column, and to also color that column 'blue'. Unless you start with a clean slate, you will have to update View Manager and the Look&Feel color rules yourself.
Version 2.0.6 - 2015-12-18 Consolidation now properly updates the index for Series-Like custom columns.
Version 2.0.5 - 2015-12-17 Miscellaneous technical changes.
Version 2.0.4 - 2015-12-15 Miscellaneous technical changes and enhancements.
Version 2.0.3 - 2015-12-11 Miscellaneous tweaks.
Version 2.0.2 - 2015-12-02 Menu function to delete "Non-Library Codes Identifiers" changed to also not delete new LC Identifier "loc_lccn".
Version 2.0.1 - 2015-11-20 Metadata Tools "Delete non-ISBN/ISSN Identifiers" function changed to also not delete any Identifiers created by the Library Codes plugin (viaf,lc,lccn,isni, oclc, oclc-owi). The pushbutton and other text were changed to say "Non-Library Codes".
Version 2.0.0 - 2015-10-30 Major Functionality and User Interface Enhancements to Enable the Consolidation of all, some, or no Source Library Custom Columns. Performance Improvements. Revised ToolTips and Embedded Instructions and FAQs.
Version 1.0.0 - 2015-10-02 Initial release.

[DaltonST]

Reserved for future use.

[DaltonST]

This is reserved for future use.

[Tanjamuse]

I would like to vote for a custom column update

[Terisa de morgan]

I would be happy with series

[DaltonST]

@Tanjamuse,

Already in process. As the OP says:

[4] 'Standard Metadata' does not include 'Custom Columns'. Are there any plans to also consolidate all arbitrary and random 'Custom Columns' from all Source Libraries into the snapshot Target Library?

Yes; it is planned for a future release.

I am currently working on Release 2.0.0, which dynamically builds custom columns in the Target Library for the set of all identical custom columns in all of the Source Libraries, then maps all Source Library custom columns to those Target Library custom columns. You will be able to easily say "map all of them" or "map none of them". Or, by manually updating a new table, you can pick and choose for each Source Library custom column which you want to map to the Target Library and which you do not want mapped. If not mapped, they will be ignored by the Consolidation Job.

The definition of "identical" means that the rows in the Source Library metadata.db table custom_columns are identical in every respect (except for the custom column id number, of course). So, it would be a good idea to standardize your custom columns in every respect across all of your Source Libraries. That includes the Description that you see when you hover your mouse over the GUI column heading, and not just the datatype (text or comments or ...) plus the rest that you specify when you create a new custom column in "Calibre > Preferences > Add Your Own Columns".

I have finished testing everything except actually populating the Target Library custom columns from their mapped Source Library custom columns in the revised Consolidation Job, which is the easy part. The hard part was the functionality that the users can see and touch.

Release 2.0.0 is currently on track for a release next week, although as I do this purely for my personal use, amusement and pleasure, it may be sooner or it may be later than next week.



DaltonST

[DaltonST]

@Terisa de Morgan:

Release 1.0.0 already does Series, since it is standard metadata. It takes a snapshot of everything except for the ebook itself and custom columns, and the latter is under development.

Attached is a screen-snip of the CALM Target Library in the GUI that shows Series.


DaltonST


[n.b.: CALM Consolidates a snapshot of all of the standard metadata for all of your Calibre Libraries into a temporary CALM Target Library. CALM's Target Library is a valid Calibre Library. Any plug-in that works in Release 2.40 of Calibre works on the CALM Target Library, because it is a Calibre Library]

[Terisa de morgan]

Great @DaltonST, as it's my main problem (tags are easily standardized).

[Tanjamuse]

Would it be possible to add more than the title to search for duplicates? Or maybe it could work in tandem with the "Find Duplicate" plugin?

[DaltonST]

Originally Posted by Tanjamuse

Would it be possible to add more than the title to search for duplicates? Or maybe it could work in tandem with the "Find Duplicate" plugin?

@Tanjamuse:

I think you are missing something. CALM's Target Library is a valid Calibre Library. Any plug-in that works in Release 2.40 of Calibre works on the CALM Target Library, because it is a Calibre Library.

I guess you have not installed it, which would explain your question.



DaltonST

[Terisa de morgan]

I'm not able to define any source library. All the time is "No libraries were selected". I'm running calibre at windows and I've defined the path with "/" and "\", and with and without metadata.db.

[DaltonST]

@Terisa de morgan:

You do not need to manually define them by keying in anything. Simply click the "Load All Libraries" button, and it will load all of the Calibre Libraries that Calibre knows about, sorted from most-used to least-used. You simply check the ones you want to consolidate, then click the "Erase Unchecked Sources" button, then click the "Save & Reload" button.

If you really want to do manual work, your paths must be like this: X:/Calibre/Normal1


Quick Switch in Calibre to a recent Library that you have recently used in Calibre, and run CALM from there. As the embedded instructions in the plugin on the Target Tab say, you cannot do anything in the first 2 Tabs while in the CALM Target Library anyway. Be sure to read the instructions on the Target Tab before proceeding.


DaltonST

[Terisa de morgan]

The button "Load all" and, mainly, the button "save" in the library tab (not the main button) did the trick, thank you.

[DaltonST]

Version 2.0.0 - 2015-10-30: Major Functionality and User Interface Enhancements to Enable the Consolidation of all, some, or no Source Library Custom Columns. Performance Improvements. Revised ToolTips and Embedded Instructions and FAQs.


The CALM Target Library is the "target" of the Consolidation process, and is a fully functional Calibre Library that supports all Calibre plugins that support Calibre Version 2.40 and above.


Essential plugin if you use CALM: View Manager


Recommended plugin if you use CALM: MultiColumnSearch




DaltonST

[Tanjamuse]

Thank you so much. I can't wait to try it out.