Linux: Automatic rsync snapshot of the calibre libraries

[Adoby]

I suspect there are many that use rsync in weird and wonderful ways to backup their calibre libraries.

There are two reasons why it is nice to use rsync to make snapshots.

The first reason is that it syncs. So only modified files have to be copied. This means that it is very fast and efficient.

The second reason is that rsync can create hard links to unchanged files, so each snapshot still looks like a full backup, but unchanged files are only copied once. This means that you can have a very large amount of snapshots, and they don't take up much more space than they really need. No need to store duplicate files. Each file only has to be copied if it changes. The hard links are to previous copies in the snapshots. And these hard linked files use a copy counter, so you can freely delete snapshots. Only when you delete the last copy is that file really deleted.

Here is how I do it. (Ubuntu 14.04.01)

~/bin/snapshot_calibre.sh
This executable script is a general "snapshotter" that I have written and configured to make snapshots of my calibre libraries. It has to be edited to specify where the calibre libraries are and where the snapshots are to be stored, and for how many days you want to keep snapshots.

This script can easily be copied and changed to make snapshots of other folders as well. I also use it to make snapshots of my documents folder. It is then set up to make a new snapshot every time I start the computer.

It is best if you have the snapshot on an other drive, or even on another computer. I have all my snapshots on my NAS.

If you need to use a snapshot to restore lost data, copy the entire snapshot before you start changing or editing anything. Otherwise you may mess up other snapshots.

~/bin/calibre
This is also a script that is executable. All it does is launch /opt/calibre/calibre and then when calibre is exited, it immediately runs ~/bin/snapshot_calibre.sh. The nice thing about this is that commands in ~/bin takes precedence over other commands. So there is no need to change anything else to automate the snapshots. The normal launcher in Unity will use this script and updates of calibre will not overwrite anything. I haven't tested this trick in other environments...

One cause of problem may be if you start and exit calibre quickly several times, before snapshot_calibre.sh has had time to sync. Especially the first time you sync, when the sync can take a while. Just don't do it.


~/bin/snapshot_calibre.sh

Spoiler:

#!/bin/bash
# General folder snapshot maker. Makes snapshots of calibre libraries.
# Makes use of rsync and hardlinks to create snapshots that look like full backups.
# Don't use or update files in a snapshot. Copy the snapshot first.
# Deletes snapshots older than a certain number of days. (30)

snapshot_path='/path to where you keep your calibre library snapshots'
source_path='/path to where you keep your calibre library'
age_snapshot='30'
snapshot_name=$(date +%F_%T)

# Delete earlier failed attempt at making a snapshot, if any
rm -rf "$snapshot_path"/unfinished

# Find latest snapshot, if any
latest_snapshot=$(ls -rt "$snapshot_path" | tail -1)

# Create folder for this snapshot, unfinished so far...
mkdir "$snapshot_path/unfinished"

# Grab new snapshot. Use rsync to create hard links to files already in latest snapshot(s).
rsync -a --delete \
--link-dest="$snapshot_path"/"$latest_snapshot" \
"$source_path" \
"$snapshot_path"/unfinished

# Finished!
if [ $? -eq 0 ]; then
mv "$snapshot_path"/unfinished "$snapshot_path"/"$snapshot_name"
else
echo "Snapshot failed. Check free space and try again."
fi

# Delete snapshots older than $age_snapshot
find $snapshot_path -maxdepth 1 -type d -mtime +$age_snapshot -print0 | xargs -0 rm -rf

~/bin/calibre

Spoiler:

#!/bin/bash
# calibre launcher that takes a snapshot of the calibre libraries.
# Call it just "calibre" and place in ~/bin and make executable
# And make sure snapshot_calibre.sh is set up correctly.
/opt/calibre/calibre
snapshot_calibre.sh

How do you use rsync?

[eschwartz]

I'd do much the same... (but I don't feel the need for backups, ATM. I do use rsync to dropbox the libraries, though.)

Why don't you explicitly kill the snapshot process or do some sort of check with a lockfile, before launching /opt/calibre/calibre

[Adoby]

Originally Posted by eschwartz

I'd do much the same... (but I don't feel the need for backups, ATM. I do use rsync to dropbox the libraries, though.)

Why don't you explicitly kill the snapshot process or do some sort of check with a lockfile, before launching /opt/calibre/calibre

I've never had any problem with this. So I never bothered to add any lockfile or similar. Usually a sync is done in seconds. And even if I relaunch calibre quickly, the sync can continue while calibre is launched and while I use it. I just thought I should mention it as a potential problem.

[eschwartz]

Originally Posted by Adoby

I've never had any problem with this. So I never bothered to add any lockfile or similar. Usually a sync is done in seconds. And even if I relaunch calibre quickly, the sync can continue while calibre is launched and while I use it. I just thought I should mention it as a potential problem.

Where's your sense of completeness?

*mentions it as a problem but doesn't even care enough to bother himself, hmph. muttermuttermutter...*

[Adoby]

OK!

I accepted the implied challenge and improved the script.

Now I use flock to ensure that only one instance of the script can run, use zenity to provide some feedback while the snapshot is written by rsync and write a small log to each snapshot folder. Also it is now all in one file, this is no longer a general folder snapshotter that is used to take snapshots of the calibre libraries.

It is now a calibre launcher script that also takes a snapshot after calibre exits.

Both flock and zenity are installed by default in Ubuntu 14.04.01, and also in many other dists, I think. So typically no need to do anything other than fill in the correct paths, place the script in the file ~/bin/calibre and make it executable.

~/bin/calibre

Spoiler:

#!/bin/sh
# calibre launcher that also takes a snapshot of the calibre libraries. 
# Call it just "calibre" and place in ~/bin and make executable
# Makes use of rsync and hardlinks to create snapshots that look like full backups. 
# Don't use or update files in a snapshot. Copy the snapshot first.
# Deletes snapshots older than a certain number of days.
# Uses flock(1) to ensure only one instance of this script can run.
# Uses zenity(1) to provide progress dialog and error message if needed.
# Both flock and zenity are installed by default in Ubuntu 14.04.

# Change these variables to customize:
snapshot_path='/full/path/to place to store snapshots'
source_path='/full/path/to place where the calibre library/libraries are'
age_snapshot='30'
snapshot_name=$(date +%F_%T)

# Use subshell and flock to ensure only one instance of this script can run
(
	# Exit if script is already running
	flock -n 9 || exit 1

 	# Run calibre
	/opt/calibre/calibre

	# Run the majority of the script in a subshell, and send the output from that subshell to zenity
	(
		# Output text to zenity progress dialog
		echo "10\n# Preparing for snapshot, please wait..."

		# Delete earlier failed attempt at making a snapshot, if any
		rm -rf "$snapshot_path"/unfinished		

		# Find latest snapshot, if any
		latest_snapshot=$(ls -rt "$snapshot_path" | tail -1)

		# Create folder for this snapshot, unfinished so far...
		mkdir "$snapshot_path"/unfinished

		# Grab new snapshot. Use rsync to create hard links to files already in latest previous snapshot(s).

		echo "40\n# Taking snapshot with rsync, please wait..."

		rsync -a --delete --stats \
		      --link-dest="$snapshot_path"/"$latest_snapshot" \
		      "$source_path" \
		      "$snapshot_path"/unfinished > "$snapshot_path"/unfinished/rsync.log

		# Finished!
		if [ $? -eq 0 ]; then
			mv "$snapshot_path"/unfinished "$snapshot_path"/"$snapshot_name"
		else
			exit 1
		fi

		# Delete snapshots older than $age_snapshot

		echo "# Deleting old snapshots, please wait..."

		find $snapshot_path -maxdepth 1 -type d -mtime +$age_snapshot -print0 | xargs -0 rm -rf

		echo "99\n# Snapshot done! Check rsync.log in snapshot folder."
		sleep 2
		echo "100"
	) |
	zenity --progress --auto-close --pulsate \
	  --title="Taking snapshot of calibre library" \
	  --text="Please wait..." \
	  --percentage=1

	# Snapshot failed?
	if [ "$?" = 1 ] ; then
	        zenity --error --text="Snapshot failed."
	fi

) 9>/var/lock/snapshot_calibre_lock