Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 08-11-2014, 01:23 PM   #1
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,105
Karma: 5168844
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Ubuntu Linux, Cybook Opus, Motorola Xoom with Mantano Premium
Linux: Automatic rsync snapshot of the calibre libraries

I suspect there are many that use rsync in weird and wonderful ways to backup their calibre libraries.

There are two reasons why it is nice to use rsync to make snapshots.

The first reason is that it syncs. So only modified files have to be copied. This means that it is very fast and efficient.

The second reason is that rsync can create hard links to unchanged files, so each snapshot still looks like a full backup, but unchanged files are only copied once. This means that you can have a very large amount of snapshots, and they don't take up much more space than they really need. No need to store duplicate files. Each file only has to be copied if it changes. The hard links are to previous copies in the snapshots. And these hard linked files use a copy counter, so you can freely delete snapshots. Only when you delete the last copy is that file really deleted.

Here is how I do it. (Ubuntu 14.04.01)

~/bin/snapshot_calibre.sh
This executable script is a general "snapshotter" that I have written and configured to make snapshots of my calibre libraries. It has to be edited to specify where the calibre libraries are and where the snapshots are to be stored, and for how many days you want to keep snapshots.

This script can easily be copied and changed to make snapshots of other folders as well. I also use it to make snapshots of my documents folder. It is then set up to make a new snapshot every time I start the computer.

It is best if you have the snapshot on an other drive, or even on another computer. I have all my snapshots on my NAS.

If you need to use a snapshot to restore lost data, copy the entire snapshot before you start changing or editing anything. Otherwise you may mess up other snapshots.

~/bin/calibre
This is also a script that is executable. All it does is launch /opt/calibre/calibre and then when calibre is exited, it immediately runs ~/bin/snapshot_calibre.sh. The nice thing about this is that commands in ~/bin takes precedence over other commands. So there is no need to change anything else to automate the snapshots. The normal launcher in Unity will use this script and updates of calibre will not overwrite anything. I haven't tested this trick in other environments...

One cause of problem may be if you start and exit calibre quickly several times, before snapshot_calibre.sh has had time to sync. Especially the first time you sync, when the sync can take a while. Just don't do it.


~/bin/snapshot_calibre.sh

Spoiler:
#!/bin/bash
# General folder snapshot maker. Makes snapshots of calibre libraries.
# Makes use of rsync and hardlinks to create snapshots that look like full backups.
# Don't use or update files in a snapshot. Copy the snapshot first.
# Deletes snapshots older than a certain number of days. (30)

snapshot_path='/path to where you keep your calibre library snapshots'
source_path='/path to where you keep your calibre library'
age_snapshot='30'
snapshot_name=$(date +%F_%T)

# Delete earlier failed attempt at making a snapshot, if any
rm -rf "$snapshot_path"/unfinished

# Find latest snapshot, if any
latest_snapshot=$(ls -rt "$snapshot_path" | tail -1)

# Create folder for this snapshot, unfinished so far...
mkdir "$snapshot_path/unfinished"

# Grab new snapshot. Use rsync to create hard links to files already in latest snapshot(s).
rsync -a --delete \
--link-dest="$snapshot_path"/"$latest_snapshot" \
"$source_path" \
"$snapshot_path"/unfinished

# Finished!
if [ $? -eq 0 ]; then
mv "$snapshot_path"/unfinished "$snapshot_path"/"$snapshot_name"
else
echo "Snapshot failed. Check free space and try again."
fi

# Delete snapshots older than $age_snapshot
find $snapshot_path -maxdepth 1 -type d -mtime +$age_snapshot -print0 | xargs -0 rm -rf


~/bin/calibre
Spoiler:
#!/bin/bash
# calibre launcher that takes a snapshot of the calibre libraries.
# Call it just "calibre" and place in ~/bin and make executable
# And make sure snapshot_calibre.sh is set up correctly.
/opt/calibre/calibre
snapshot_calibre.sh



How do you use rsync?

Last edited by Adoby; 08-11-2014 at 01:56 PM.
Adoby is online now   Reply With Quote
Old 08-11-2014, 02:19 PM   #2
eschwartz
Irrational Optimist
eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.
 
Posts: 6,483
Karma: 10683802
Join Date: Nov 2012
Location: The Beaten Path, USA, Roundworld, This Side of Infinity
Device: Kindle Touch (Wifi only)
I'd do much the same... (but I don't feel the need for backups, ATM. I do use rsync to dropbox the libraries, though.)

Why don't you explicitly kill the snapshot process or do some sort of check with a lockfile, before launching /opt/calibre/calibre

Last edited by eschwartz; 08-11-2014 at 02:21 PM.
eschwartz is online now   Reply With Quote
Old 08-11-2014, 02:36 PM   #3
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,105
Karma: 5168844
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Ubuntu Linux, Cybook Opus, Motorola Xoom with Mantano Premium
Quote:
Originally Posted by eschwartz View Post
I'd do much the same... (but I don't feel the need for backups, ATM. I do use rsync to dropbox the libraries, though.)

Why don't you explicitly kill the snapshot process or do some sort of check with a lockfile, before launching /opt/calibre/calibre
I've never had any problem with this. So I never bothered to add any lockfile or similar. Usually a sync is done in seconds. And even if I relaunch calibre quickly, the sync can continue while calibre is launched and while I use it. I just thought I should mention it as a potential problem.
Adoby is online now   Reply With Quote
Old 08-11-2014, 03:25 PM   #4
eschwartz
Irrational Optimist
eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.
 
Posts: 6,483
Karma: 10683802
Join Date: Nov 2012
Location: The Beaten Path, USA, Roundworld, This Side of Infinity
Device: Kindle Touch (Wifi only)
Quote:
Originally Posted by Adoby View Post
I've never had any problem with this. So I never bothered to add any lockfile or similar. Usually a sync is done in seconds. And even if I relaunch calibre quickly, the sync can continue while calibre is launched and while I use it. I just thought I should mention it as a potential problem.
Where's your sense of completeness?

*mentions it as a problem but doesn't even care enough to bother himself, hmph. muttermuttermutter...*
eschwartz is online now   Reply With Quote
Old 08-13-2014, 05:24 AM   #5
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,105
Karma: 5168844
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Ubuntu Linux, Cybook Opus, Motorola Xoom with Mantano Premium
OK!

I accepted the implied challenge and improved the script.

Now I use flock to ensure that only one instance of the script can run, use zenity to provide some feedback while the snapshot is written by rsync and write a small log to each snapshot folder. Also it is now all in one file, this is no longer a general folder snapshotter that is used to take snapshots of the calibre libraries.

It is now a calibre launcher script that also takes a snapshot after calibre exits.

Both flock and zenity are installed by default in Ubuntu 14.04.01, and also in many other dists, I think. So typically no need to do anything other than fill in the correct paths, place the script in the file ~/bin/calibre and make it executable.

~/bin/calibre
Spoiler:

Code:
#!/bin/sh
# calibre launcher that also takes a snapshot of the calibre libraries. 
# Call it just "calibre" and place in ~/bin and make executable
# Makes use of rsync and hardlinks to create snapshots that look like full backups. 
# Don't use or update files in a snapshot. Copy the snapshot first.
# Deletes snapshots older than a certain number of days.
# Uses flock(1) to ensure only one instance of this script can run.
# Uses zenity(1) to provide progress dialog and error message if needed.
# Both flock and zenity are installed by default in Ubuntu 14.04.

# Change these variables to customize:
snapshot_path='/full/path/to place to store snapshots'
source_path='/full/path/to place where the calibre library/libraries are'
age_snapshot='30'
snapshot_name=$(date +%F_%T)

# Use subshell and flock to ensure only one instance of this script can run
(
	# Exit if script is already running
	flock -n 9 || exit 1

 	# Run calibre
	/opt/calibre/calibre

	# Run the majority of the script in a subshell, and send the output from that subshell to zenity
	(
		# Output text to zenity progress dialog
		echo "10\n# Preparing for snapshot, please wait..."

		# Delete earlier failed attempt at making a snapshot, if any
		rm -rf "$snapshot_path"/unfinished		

		# Find latest snapshot, if any
		latest_snapshot=$(ls -rt "$snapshot_path" | tail -1)

		# Create folder for this snapshot, unfinished so far...
		mkdir "$snapshot_path"/unfinished

		# Grab new snapshot. Use rsync to create hard links to files already in latest previous snapshot(s).

		echo "40\n# Taking snapshot with rsync, please wait..."

		rsync -a --delete --stats \
		      --link-dest="$snapshot_path"/"$latest_snapshot" \
		      "$source_path" \
		      "$snapshot_path"/unfinished > "$snapshot_path"/unfinished/rsync.log

		# Finished!
		if [ $? -eq 0 ]; then
			mv "$snapshot_path"/unfinished "$snapshot_path"/"$snapshot_name"
		else
			exit 1
		fi

		# Delete snapshots older than $age_snapshot

		echo "# Deleting old snapshots, please wait..."

		find $snapshot_path -maxdepth 1 -type d -mtime +$age_snapshot -print0 | xargs -0 rm -rf

		echo "99\n# Snapshot done! Check rsync.log in snapshot folder."
		sleep 2
		echo "100"
	) |
	zenity --progress --auto-close --pulsate \
	  --title="Taking snapshot of calibre library" \
	  --text="Please wait..." \
	  --percentage=1

	# Snapshot failed?
	if [ "$?" = 1 ] ; then
	        zenity --error --text="Snapshot failed."
	fi

) 9>/var/lock/snapshot_calibre_lock
Adoby is online now   Reply With Quote
Reply

Tags
automated, backup, linux, rsync, snapshots

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Fully automatic calibre installer/updater for linux eschwartz Related Tools 17 10-02-2014 03:19 PM
Tutorial linux backup of Kindle, annotations/pagenumbers/stuff with rsync eschwartz Kindle Developer's Corner 7 03-29-2014 05:45 PM
Does Calibre Keep a Snapshot of Device Contents? Jimbo724 Devices 3 09-12-2013 09:29 AM
How to easily backup all your notes, marks & reading positions with rsync under Linux Dylan Tomorrow Amazon Kindle 5 10-26-2012 03:18 PM
Missing libraries in calibre binary release for Linux davide125 Calibre 5 09-02-2011 07:53 AM


All times are GMT -4. The time now is 08:44 PM.


MobileRead.com is a privately owned, operated and funded community.