Is there a way to "sync" two libraries?

[LadyKate]

I've been wondering what is the best way to have a duplicate library on my external drive for backup purposes.

I have never used any incremental backup for my calibre libraries but find that my library is too large to routinely use a complete backup now.

My current method has been to copy the library to the external but it takes too long to do it completely to be good for daily or even weekly backup. As staying away from the file structure is ingrained as a "no no" I wondered how people handle doing this.

Suggestions are welcome.

[BetterRed]

@LadyKate - quite a few folks here use Free File Synch in mirror mode. I use a similar program Goodsynch, at the time of selecting it FFS was in its infancy.

I normally run it at shutdown, takes a few minutes to sync my data drive: Calibre Libraries, Documents, Pictures, Music etc, plus the calibre, Handbrake, Irfanview etc folders in %APPDATA%\ etc.

BR

[LadyKate]

That sounds like an option. Is the backup library totally usable?

I wonder if perhaps in the future we might see a "sinc library" feature.

[BetterRed]

Originally Posted by LadyKate

That sounds like an option. Is the backup library totally usable?

I wonder if perhaps in the future we might see a "sinc library" feature.

@LadyKate - yep, you can point calibre at the mirrored library folder. There's no packaging into something obscure - the backup mirror looks exactly like the source.

Speaking for myself I'd never use an application specific backup feature. I prefer to have one process using one backup application for all my data; ebooks, photos, music, investment portfolio, tax records, correspondence... et cetera.

BR

[LadyKate]

I'm curious, does the backup work properly if the calibre library is open when the backup occurs?

[eschwartz]

Well, if calibre is busy moving files around, the backup might catch the library halfway through, and have some files in the wrong place.

It depends on what sync application you use.
FreeFileSync (Windows) in Admin mode can use Volume Shadow Copy I believe to get a view of the filesystem at a single moment in time, then make a copy of that.
On linux, you can use an LVM snapshot to do much the same thing.


calibre performs all file writes in an atomic fashion, so you shouldn't get corrupted files.


I still wouldn't risk it, either way. I prefer scripting the backup to happen automatically after calibre closes, and locking calibre from opening while it is running.

[BetterRed]

Originally Posted by eschwartz

I still wouldn't risk it, either way. I prefer scripting the backup to happen automatically after calibre closes, and locking calibre from opening while it is running.

Calibre might also be 'lazy' writing opf files as a result of a bulk change

Originally Posted by LadyKate

I'm curious, does the backup work properly if the calibre library is open when the backup occurs?

@LadyKate - I run the backup at shutdown. First step is to run calibre --shutdown-running-calibre, then run calibredb backup_metadata against each library (to flush pending opf writes), and then it does the incremental backup using GoodSych (similar to Free File Sync).

Rarely takes more than 15 minutes, and that includes all my data, and 'critical' control data such as windows registry, calibre config, firefox profiles etc.

I also run the 'job' manually before allowing Windows Update to do its stuff.

BR

[LadyKate]

Ok, I shut down calibre before doing a mirror backup of my library directory but find I have a directory RecycleBin.ffs_tmp for some reason that I can't figure out the reason for.

Other than that it appears I have a duplicate directory of my library with the one additional directory.

Thank you for your help.

[eschwartz]

Whatever it is, it is probably something to do with FFS itself, based on the directory name.

[AnotherCat]

Originally Posted by LadyKate

Ok, I shut down calibre before doing a mirror backup of my library directory but find I have a directory RecycleBin.ffs_tmp for some reason that I can't figure out the reason for.

Other than that it appears I have a duplicate directory of my library with the one additional directory.

Thank you for your help.

If you have files going to the Recycle Bin in the FFS synchronization settings RecycleBin.ffs_tmp is created. It is normally deleted at the end of the session but may stay if there is some reason such as a file in it that is in use in another application.

Its non deletion is of no consequence and should just disappear next time you run FFS.