speed up metadata backup

[ilovejedd]

Is there a way to speed up metadata backup? I'm in the process of backing up my library so I selected the "Start backing up metadata of all books" to update the opf files but it's taking forever to finish (1 book every 3 sec per the dialog). I know the opf files it creates are just failsafes in case metadata.db gets corrupted nevertheless, I'd like to have those updated before I backup my Calibre library. Thanks!

[itimpi]

OPF files are normally updated any time you press the OK button on the Edit Metadata dialogs, so you should not normally need to run the option to get them backed up.

That is intended primarily for use on old libraries that pre-date the existence of the OPF files in a Calibre library, and IS deliberately designed to run at a limited pace so that the function could be run as a background job without severely affecting performance of Calibre in the foreground.

[ilovejedd]

I changed some templates around in the custom columns and those haven't been propagated to the OPF backups. I know that generating the OPF is usually done in the background and is designed to use as little system resource as possible but in my case (multiple libraries with 500-15,000 ebooks each), I would prefer to have the option to run it in the foreground (or have a CLI utility that can do it) and practically lock up my system in exchange for finishing the task much faster.

[kovidgoyal]

calibre-debug -c "from calibre.library import db; db = db('/path/to/library'); db.dump_metadata(db.all_ids())"

[ilovejedd]

Originally Posted by kovidgoyal

calibre-debug -c "from calibre.library import db; db = db('/path/to/library'); db.dump_metadata(db.all_ids())"

Awesome, thanks!

[chaley]

It is worth noting that as of the latest release (or thereabouts), composite custom column templates are saved in the metadata_db_prefs_backup.json file every time you exit the calibre gui. This file is used by restore to rebuild the custom columns, as well as to restore a lot of other things like saved searches, user categories, plugboards, grouped search terms, custom template functions, and several library-level preferences.

The custom column descriptions in the metadata.opf files are used only if the metadata_db_prefs_backup.json is missing or corrupted.

[DaltonST]

Originally Posted by kovidgoyal

calibre-debug -c "from calibre.library import db; db = db('/path/to/library'); db.dump_metadata(db.all_ids())"

Using the 64 byte Win 7 version of Calibre 1.0.0, I receive the following errors whenever I try to use calibre-debug to dump the .opf files for my 26GB 32,000 epub library using Kovid's code snippet from this thread. I have tried both with and without Calibre running simultaneously, but to no avail. Any ideas as to the cause of the SHLock error? Thanks in advance for your help.

C:\Program Files\Calibre2>calibre-debug.exe -c "from calibre.library import db;
db = db('C:\Users\Public\Public eBooks\Calibre2'); db.dump_metadata(db.all_ids()
)"

Python function terminated unexpectedly
can't downgrade SHLock object (Error Code: 1)
Traceback (most recent call last):
File "site.py", line 132, in main
File "site.py", line 109, in run_entry_point
File "site-packages\calibre\debug.py", line 231, in main
File "<string>", line 1, in <module>
File "site-packages\calibre\db\legacy.py", line 348, in dump_metadata
File "site-packages\calibre\db\cache.py", line 55, in ans
File "site-packages\calibre\db\cache.py", line 1012, in dump_metadata
File "site-packages\calibre\db\legacy.py", line 146, in all_ids
File "site-packages\calibre\db\cache.py", line 54, in ans
File "site-packages\calibre\db\locking.py", line 212, in __enter__
File "site-packages\calibre\db\locking.py", line 83, in acquire
File "site-packages\calibre\db\locking.py", line 153, in _acquire_shared
calibre.db.locking.LockingError: can't downgrade SHLock object

[kovidgoyal]

For version 1.0 and newer, use this code instead:

calibre-debug -c "from calibre.library import db; db = db('/home/kovid/test library'); db.dump_metadata(tuple(db.all_ids()))"