Vast rewrites when importing a wrongly cased author

[sciurius]

When I have an author (e.g., "Asimov, Isaac") in my library, and I import a book that has the same author name but with differing case, e.g. "ASIMOV, Isaac", Calibre interprets this as an author name change and updates the filenames on disk and all existing opf files of books of that author.

This can make importing a folder of books pretty slow.

Is there a way to avoid this?

Running Calibre 2.78.0 on Fedora 25.

[theducks]

2.78 has a cane and a beard (btw Kovid never updates past Major releases.)

Calibre is an RDB, the case change is written into 1 location (Authors table), The Books Table use the Index value from the Authors table. No change for all existing, 1 new entry for the currently being imported book.

It is unfortunate, that calibre updates ANY fields case. But, that is the way it works, from almost day one.

[kovidgoyal]

Renaming files is pretty fast, it shouldn't really bottleneck things (it works by hard linking and then deleting the originals, in a transactional manner). And as theducks said, updating the database is of no consequence, since it only needs to update a single row for a case change.

[sciurius]

It is the rewriting of all the opf files that takes the time.

[theducks]

Originally Posted by sciurius

It is the rewriting of all the opf files that takes the time.

But that is a background task that happens as cycles permit.
If you close Calibre, it will continue the next session.
Besides, the OPF in question is a BACKUP of the DB entry

[sciurius]

As far as I can judge, it feels synchronous to me.

When in Calibre, I change in the autor column the case of one of the letters of a name of an artist with several books, I hear a lot of disk activity (all the opf files being rewritten) and Calibre is non-responsive until this completes.

Do I have one of my settings wrong?

[BetterRed]

@Scirius - On my Win 10 system the OPF writes occur at about 1/sec, with little effect on calibre or general performance. That's been true on 3 different computers running Windows and Mint since I first installed calibre (0.8.??) in early 2012.

The only settings I can think that might, and its a pretty big might, have a bearing on the issue are

• the Maximum [. . .] jobs and the Limit [. . .] jobs [. . .] settings in Preferences->Miscellaneous -- mine are 1 and unchecked;
• the Job Priority setting in Preferences->Miscellaneous -- mine is Low.

My 'gripe' with this issue is not the writing of them but the fact they then have to be copied by my incremental backups, which not only slows it down, but makes eyeball checks of the backup logs harder (wood for trees). But I don't have a alternative solution to database rebuilding - apart from mirroring the database. And the OPF files are used for other purposes so... I think we're stuck with it

In my backup scripts, I have a set of calibredb backup_metadata commands to flush any pending OPF writes before the actual backup task is started.

BR

[DaltonST]

A good way to avoid that issue is to not import raw books directly into your Main Library, but rather into a Workbench Library. Scrub, standardize and add missing metadata to the books while still in your Workbench Library. Such activity has zero impact on the books already in your Main Library. Finally, move your new books from your Workbench Library to your Main Library.

If your Main Library is actually fairly trashed-out, then rename it Workbench, and create a fresh Main. Move only the clean books from Workbench to Main, and work on the remaining books now in your Workbench.

Be sure to Vacuum the metadata.db file, since it grows to an enormous size over time. Do so with Library > Library Maintenance > Check Library, which first vacuums metadata.db and then compares it to your physical books, and vice-versa.

If you do not want any .opf files being created while you make many mass-changes in your Workbench Library, you can purge the queue for just the WorkBench Library using the Job Spy plugin, which has a Tool to do just that (highlighted in the attachment).

That tool is designed to be used when it is a total waste of cpu and disk to create .opf files at a rate of 1 per second constantly for ever-changing metadata that is actively being worked on in a Workbench Library.

When you move your clean books to your Main Library, a final .opf file will be created for that new book in that Library.

Obviously, do not purge the queue for your Main Library. Those .opf files allow you to recover from a corrupt metadata.db file. The exception to that rule might be creating a new Custom Column that is easily recomputed, and you don't want 50,000 .opf files being created just to be able to restore a Custom Column that is easily re-populated on-demand (e.g. from a simple, fast job).





DaltonST

[kovidgoyal]

OPF writing is not synchronous. And it is rate limited.

[BetterRed]

@DaltonST - you are assuming that all metadata is known at the time a book is added to a library, with fiction that's probably true, but with non-fiction less so. Especially with subject tags.

Imagine a passing reference in a biography of famous person A to person B who later becomes also (in)famous. FX: I wonder how many Hollywood bios have a passing mention of HW, someone might have a "Mentioned" column they want to update.

Or an obscure financial instrument confected in the 1970s that becomes very relevant 30 years later. Who knew the role that CoCo bonds would play on Wall Street in the 2007/8 crash, or what a CDS was before the Greek debt crisis struck.

A full text X1 search for 'Coco Bond', 'Contingent Convertible Bond' or 'Enhanced Capital Note' on one of my libraries spat out a results list of ~450 documents - thanks to one of my favourite plugins I was able to tag them with 'CoCo Bonds' in the blink of an eye

BR

[Terisa de morgan]

Originally Posted by BetterRed

A full text X1 search for 'Coco Bond', 'Contingent Convertible Bond' or 'Enhanced Capital Note' on one of my libraries spat out a results list of ~450 documents - thanks to one of my favourite plugins I was able to tag them with 'CoCo Bonds' in the blink of an eye

BR

Which one? Do tell

[BetterRed]

This one ==>> Drop Search Results

BR

[Terisa de morgan]

Originally Posted by BetterRed

This one ==>> Drop Search Results

BR

I'll check, thank you

[sciurius]

Using inotify monitoring I found that the opf writing is indeed asynchronous.
The "hard linking and then deleting the originals, in a transactional manner" is the process that causes the disk activity.

A rough timing yields 3-4 renames per second (internal SATA, WD disk, Raid 1). In other words, if I have 100 books with author "Unknown", and I change the author of one book to "unknown", Calibre is unresponsive for 20-30 seconds.

It is not a big deal, I can live with this. I just wanted to make sure this is normal behaviour and I'm not doing anything wrong.

[kovidgoyal]

Wow 3-4 renames per second is pretty terrible performance. Are you sure the hardlinking is being done and not a file copy (if the hardlinking fails, calibre falls back to plain file copies).