[GUI Plugin] Consolidate All Library Metadata (CALM)

[Tanjamuse]

I actually stopped it, because I thought that it was frozen. It hadn't finished any library and it was only 2 libraries. I think it's 35 custom columns.

5400 rpm & 16 gb ram

[DaltonST]

What did the Job Log say? What in the Job Log made you think it was frozen?

[Tanjamuse]

Aside from 1% in 91 minutes?

The test libraries I've ued it took ½-1 minute.

Is there anyway in calibre I can see old jobs? Otherwise I'll run it again.

[DaltonST]

When CALM submits the Consolidation Job, it automatically brings up the Job Viewer for you so you can immediately click on the CALM job and monitor it.

The job log is very detailed and tells you exactly what it has done and is doing. It also tells you numbers of books etc. Do not monitor it by percentage. Monitor it by what the log says.

The percent complete is based on the total number of libraries being consolidated. 1% says it was still processing the very first library. For 2 total, it would have jumped from 1% to 50% as soon as it had finished the first one.

[Tanjamuse]

Starting job: CALM Consolidation 
Starting CALM Consolidation 
CALM DB: D:/Calibre/Fanfiction/CALM/metadata.db 
Tue Aug 23 16:24:14 2016 
Python: Windows   CPython   2.7.9 
SQLite Version: 3.8.4    [APSW] 
PRAGMA main.locking_mode = EXCLUSIVE; 
  
Beginning 'Consolidate All Library Metadata'  
═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════ 
  
  
Attaching to New Source Library: D:/Calibre/Fanfiction/FFnet/Crime/metadata.db 
Consolidating Source Standard Metadata: Tue Aug 23 16:24:14 2016

It has been running that for 10 minutes now. I'll let it run and let you know.

[DaltonST]

@Tanjamuse:

As a benchmark, for the purposes of discussion only, assume that for Calibre to copy all of your 50,000 books from Library 1 to Library 2 would take ~ 25 hours. I have personally experienced that rate of copy when I used a HD and not a SSD.

A "real" copy not only copies the metadata, but also all of the formats and covers and metadata.opf files, and then creates the necessary directory structure. Very RAM and disk and CPU intensive.

CALM copies only the metadata. No files. No directory structure. No formats. No covers. Just the metadata from the Source Library metadata.db to the CALM Target Library metadata.db. However, a metadata.db is a physical file that has to be physically updated with a physical temporary journal file. CALM uses a lot of RAM while copying the metadata via SQLite. So, CALM too is RAM and disk and CPU intensive. Just not as much as Calibre is for the reasons mentioned above. The CALM Job Log tells you how much RAM it used per Library, and the maximum RAM used at any point within the Consolidation Job.

So, CALM will copy (as a part of Consolidation) those same metadata-only 50,000 "books" much faster than Calibre would do when executing a "real" copy.

How much faster? Depends. Certainly no faster than 10% of the 25 hours (for the purposes of discussion only) that it would take Calibre, which would equal 2.5 hours. If it takes 20% of the 25 Calibre hours, then it would take 5 hours.

You get the picture.

When I upgraded my HD to a SSD (Solid State Drive), CALM (and everything else) ran much faster.



DaltonST

[Tanjamuse]

It has been running in 105 minutes and is still only "starting".

Starting job: CALM Consolidation 
Starting CALM Consolidation 
CALM DB: D:/Calibre/Fanfiction/CALM/metadata.db 
Tue Aug 23 16:24:14 2016 
Python: Windows   CPython   2.7.9 
SQLite Version: 3.8.4    [APSW] 
PRAGMA main.locking_mode = EXCLUSIVE; 
  
Beginning 'Consolidate All Library Metadata'  
═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════ 
  
  
Attaching to New Source Library: D:/Calibre/Fanfiction/FFnet/Crime/metadata.db 
Consolidating Source Standard Metadata: Tue Aug 23 16:24:14 2016

[DaltonST]

The Job Log says it is still working on copying the standard metadata (not the custom column metadata) from the very first library.

As I said, ignore the percentage complete since that is based on the number of libraries you are consolidating.

Read my previous post about how long CALM will take compared to doing with Calibre.

105 minutes so far would be very, very reasonable for 50,000 books.



DaltonST

[Terisa de morgan]

I've installed the new version this morning and I'm getting this error:

Spoiler:

calibre, version 2.63.0
ERROR: CALM: GENERATION OF CUSTOM COLUMNS TERMINATED PREMATURELY........ERROR: >>>>FAILURE for: calibredb add_custom_column --library-path="D:/Biblioteca Kindle/CALM" --display="{\"enum_values\":[\"ARC\", \"Biblioteca\", \"Bolt.CD\", \"Bombo\", \"Comprados\", \"Ficha\", \"First Reads\", \"Kindle Unlimited\", \"Omnibus\", \"Pagina autor\", \"Regalo\", \"Serial\", \"ePubLibre\", \"libring\", \"mirc\", \"mobileread\", \"mobilism\", \"papyrefb2\"]}" ori "Origen" enumeration

and the last error message is:

Spoiler:

calibre, version 2.63.0
ERROR: Excepción no considerada: <b>AttributeError</b>:'CALMTargetTab' object has no attribute 'purge_all_enumerations'
Traceback (most recent call last):
File "calibre_plugins.consolidate_all_library_metadata. calm_dialog", line 4964, in generate_custom_columns_control
AttributeError: 'CALMTargetTab' object has no attribute 'purge_all_enumerations'

It worked yesterday with the previous version. I've got calibre 2.63.

[Tanjamuse]

My attempt at consolidation has now run in 380 minutes and it still says the same thing it did 275 minutes ago.

Starting job: CALM Consolidation 
Starting CALM Consolidation 
CALM DB: D:/Calibre/Fanfiction/CALM/metadata.db 
Tue Aug 23 16:24:14 2016 
Python: Windows   CPython   2.7.9 
SQLite Version: 3.8.4    [APSW] 
PRAGMA main.locking_mode = EXCLUSIVE; 
  
Beginning 'Consolidate All Library Metadata'  
═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════ 
  
  
Attaching to New Source Library: D:/Calibre/Fanfiction/FFnet/Crime/metadata.db 
Consolidating Source Standard Metadata: Tue Aug 23 16:24:14 2016

[DaltonST]

It is processing 50,000 books. See my previous post about that. 5 hours represents only 20% or so of how long Calibre would take to do a "real" copy.

The Job Log will tell you if something is wrong.

To allay your fears, open Windows Task Manager, and watch the Calibre process statistics change for CPU and memory. You may have to tell TM to show you all of the processes it has to see everything.


DaltonST

[DaltonST]

@Terisa de morgan:

The "purge_all_enumerations" is telling you that it was trying to purge everything to do with Custom Columns of datatype 'enumeration' because your CALM settings for them are corrupt. I will fix that, but that is just a symptom of the real problem.

Do this:

[1] Refresh the CALM Target Library.
[2] Check your Source Libraries, and Save them.
[3] Go to the Source Custom Columns Tab, and "get" all of them. Then, Deactivate all of the composite Custom Columns.
[4] Generate again.

If [3] were skipped because your Source Libraries did not change, then that is the reason. Cannot skip [3], even if the Source Libraries did not change. [3] must be done every time.



DaltonST

[DaltonST]

@Terisa de Morgan:

Wait..did you install the special .zip that was just for Tanjamuse? If so, you must download the official one.


DaltonST

[Tanjamuse]

The zip-file seems to work, but I must admit I've given up on using it on my fanfiction libraries. I find it tiresome that I have to run it every time I add new stories to one of the libraries. Also I would probably end up having to use more than one CALM library and I'm pretty sure that that's not possible.

[Terisa de morgan]

I have installed the official version... and I've never gone over steps 2 and 3 before. I always skip steps 2 and 3 and it has never failed. Yesterday either with the previous version.