Backing up metadata.db on exit

[qwertyuio]

After messing up a bulk edit I am wondering if it is possible to get calibre to create a backup copy of metadata.db each time it is shutdown. That is, on exit, copy metadata.db to metadata-backup-yymmdd.db (or some other numbering system) up to a maximum number of backups.

This would allow me to restore a previous metadata.db and run check_library to revert to a previous state. I do keep regular backups of my filesystem but these are only run once a day. When I spend an afternoon cleaning up metadata I usually start and stop calibre several times so there would be the potential to save a lot of work reverting to a known good state.

I thought of wrapping calibre in a script to do this but I have several libraries so it is a bit more complicated than a 3 line script.

[kovidgoyal]

If i were you i'd setup backup of all my data on a daily schedule including calibre libraries. This should be done once at a system level then you can forget about it.

If you really want to backup when calibre shutsdown then use a script, organize all your libraries under a single folder and backup that folder in your script after calibre quits.

[ilovejedd]

Originally Posted by qwertyuio

After messing up a bulk edit I am wondering if it is possible to get calibre to create a backup copy of metadata.db each time it is shutdown. That is, on exit, copy metadata.db to metadata-backup-yymmdd.db (or some other numbering system) up to a maximum number of backups.

This would allow me to restore a previous metadata.db and run check_library to revert to a previous state. I do keep regular backups of my filesystem but these are only run once a day. When I spend an afternoon cleaning up metadata I usually start and stop calibre several times so there would be the potential to save a lot of work reverting to a known good state.

I thought of wrapping calibre in a script to do this but I have several libraries so it is a bit more complicated than a 3 line script.

I actually run automated versioned library backups using FreeFileSync/RealTimeSync *while* Calibre is running (I leave it on 24/7).

I frequently add books remotely (via auto-add folder on Dropbox). RTS monitors the library folders for changes and after 30 minutes of inactivity, runs the backup script: one-way mirror to Dropbox and external drive plus versions on the external drive.

[Adoby]

It is possible to launch calibre from a script. Make sure the script is found first in the path of folders searched when running commands.

In the script start calibre, using the full path.

After calibre exit, continue the script with the commands needed to backup metadata.db. You may want to add a timestamp to each backup copy and delete the oldest.

I used to do this, but now I just double click a script to backup all my calibre libraries, books and database and all, using rsync snapshots. I keep 7 daily backup snapshots, 4 weekly and 4 monthly snapshots. Since unchanged files are shared among the snapshots it is very fast to update them and they take up very little storage.

I do this between two SSDs in my laptop. Usually takes less than 15 seconds. I also do this, over night, between my laptop and a NAS. Takes a couple of hours. Mostly hardlinking from old snapshots to new snapshot.

So I can travel back in time to retrieve old formats, as needed. Up to 5 months back. (4 weekly
= 1 month.)

[BetterRed]

Originally Posted by qwertyuio

After messing up a bulk edit I am wondering if it is possible to get calibre to create a backup copy of metadata.db each time it is shutdown. That is, on exit, copy metadata.db to metadata-backup-yymmdd.db (or some other numbering system) up to a maximum number of backups.

This would allow me to restore a previous metadata.db and run check_library to revert to a previous state. I do keep regular backups of my filesystem but these are only run once a day. When I spend an afternoon cleaning up metadata I usually start and stop calibre several times so there would be the potential to save a lot of work reverting to a known good state.

I thought of wrapping calibre in a script to do this but I have several libraries so it is a bit more complicated than a 3 line script.

On the basis that Prevention is better than Cure, when I'm unsure of something that I'm about to do in metadata edit search & replace, I copy some relevant books to an empty library using calibre's Copy to library tool, and then test my S&F-fu there. If I stuff up I start over.

I would advise against doing 'cherry pick' backups of calibre libraries or anything similar - e.g your browser's configuration settings. Any time 'saved' doing the backup will be lost many times over when you have to do a restore, the raison d'être of backups. Not only might it be an intricate process, it won't be a familiar process, and you're likely to be in stressed state.

Aside: The check_library tool compares the folder and file names of a library against the database and reports any 'structural' anomalies - e.g. author folders with no matching row in the database authors table, missing/extra formats etc. Some anomalies can be rectified with a button click, whilst others must be done via the file manager. But they all bare 'thinking about' before doing anything.

And if you overwrite library's metadata.db file you may need to (should ??) regenerate the metadata.opf files in the book folders (they are needed by the restore_database tool). This can be done using the calibredb backup_metadata sub command, be aware it can take a while.

BR

[theducks]

↑ ↑ ↑ ✔ Cherry picking is not advised

metadata.db NEEDS to be in Lockstep with the actual folders and formats.
IMHO use a good file sync app WHEN THE CALIBRE IS NOT RUNNING.
That will sync all Adds, changes, deletes (and skips the no changes)