linux backup of Kindle, annotations/pagenumbers/stuff with rsync

[eschwartz]

Just thought I'd share a script I use to backup my Kindle ebooks with the sidecar data. But first:


I use a custom column in calibre to make Amazon-sourced books match up to the .sdr data, so I can keep my notes/annotations even for sideloaded books and don't have to worry about syncing to Amazon etc.

The custom column (of the default type "Text, column shown in the tag browser") is named "#file", titled "Original Filename". In it, I store the filename that the Amazon book had when I added it from my Kindle, minus the file extension. It will be in the form of {book-title}_{ASIN}.

I use a custom Save Template for sending to device, (limited to the kindle2 device driver, since other devices don't need this) which looks like this:

program: 
ifempty(
	field('#file'),
	template(
		'{author_sort:sublist(0,1,&)}/{title} - {authors:sublist(0,1,&)}'
	)
)

Copy this code into the Save Template field of the device driver (see post #3 for details).

It uses the contents of custom column "file" as the savepath, if available. If not, (for instance, it isn't originally from Amazon, so I left it blank) it falls back on the main template, which I actually tweaked to only use the first author.



Now the script:

In order to preserve the .sdr/ and book data, I use an rsync script outside of calibre (saved as "$HOME/bin/kindlesync") which saves the contents of "/media/Kindle" not including the tts or index files and stuff, just the parts that might ever need to be preserved.

This also lets me restore my Kindle if it ever needs to be reset to factory conditions, or if I get a new device.

kindlesync script:

#!/bin/bash

## Set defaults

backup_folder="$HOME/Kindle Stuff/Kindle Backups/"
# These go in the backup folder
backup_excludes=.rsync-backup-exclude
restore_excludes=.rsync-restore-exclude


##### Declare usage

usage()
{
	cat <<- _EOF_
		Usage: kindlesync [OPTIONS]
		Back up contents of Kindle drive as Kindle.rsync	
		Does not include system info or TTS
		Deletes:
		    Personal Letters
		    EndActions
		    .fuse and .goutputstream garbage

		OPTIONS
		        --backup-excludes
		                        optional excludes file for backup
		                        defaults to: '$backup_excludes'
		        --restore-excludes
		                        optional excludes file for restore
		                        defaults to: '$restore_excludes'
		    -f, --folder        chooses backup folder, defaults to:
		                            '$backup_folder'
		    -b, --backup        perform backup only
		    -r, --restore       perform restore only
		    -d, --dry-run       backup, then do restore in test mode
		    -h, --help          show this usage message then exit
_EOF_
}


##### Test for interactive options

## By default, all switches turned off

backup_only=
restore_only=
dry_restore=

while [ "$1" != "" ]; do
    case $1 in
        -h|--help)          usage
                            exit
                            ;;
        -f|--folder)        shift
                            backup_folder=$1
                            ;;
		--backup-excludes)  shift
                            backup_excludes=$1
                            ;;
		--restore-excludes) shift
                            restore_excludes=$1
                            ;;
        -b|--backup)        backup_only=1
                            ;;
        -r|--restore)       restore_only=1
                            ;;
        -d|--dry-run)       dry_restore=1
                            ;;
        *)                  echo "kindlesync: unrecognized option '$1'"
                            echo "Try 'kindlesync --help' for more information."
                            exit 1
    esac
    shift
done

##### Declare Functions

do_backup()
{
    rsync -amv --exclude-from=$backup_excludes /media/Kindle/ ./Kindle.rsync
}

do_restore()
{
    rsync -amv --exclude-from=$restore_excludes --delete ./Kindle.rsync/ /media/Kindle
}

do_dry_restore()
{
    rsync -amv --dry-run --exclude-from=$restore_excludes --delete ./Kindle.rsync/ /media/Kindle
}

################################################################################################
#####                  OLD SYNC ONLY DOCUMENTS FOLDER                                ###########
#####                                                                                ###########
##### rsync -amv --exclude-from=.rsync-exclude /media/Kindle/documents/ ./documents  ###########
##### rsync -amv --delete ./documents/ /media/Kindle/documents                       ###########
################################################################################################



#### MAIN

if [ ! -d /media/Kindle/ ]; then
    echo "Your Kindle is not plugged in."
    exit 1
fi

cd "$backup_folder"

if [ "$backup_only" = "1" ]; then
    echo "Backing up to: $(pwd)/Kindle.rsync"
    do_backup
elif [ "$restore_only" = "1" ]; then
    echo "Restoring..."
    do_restore
elif [ "$dry_restore" = "1" ]; then
    echo -e "Backing up to: $(pwd)/Kindle.rsync\n\n"
    do_backup
    echo -e "Simulating restore; this isn't really happening.\n\n"
    do_dry_restore
else
    echo -e "Backing up to: $(pwd)/Kindle.rsync\n\n"
    do_backup
    echo -e "Restoring...\n\n"
    do_restore
fi

backup-excludes file:

#DOCUMENTS JUNK#
###Stupid letters I never read
- Personal\ letter*
- .fuse*
- .goutputstream-*
###I hate EndActions
- EndActions*
###Don't delete, since it may tell Amazon to download more .sdr
###But I don't need it in the backup
- *.partial

#SYSTEM#
+ /system/
+ /system/thumbnails/
+ /system/thumbnails/*
- /system/*

#TTS#
- /tts/

#Sandbox#
- /.active_content_sandbox/

#JAILBREAK#
- /acxe/
- /developer/
- /extensions/
- /linkss/
- /python/
- /usbnet/

restore-excludes file:

- /tts/
- /.active_content_sandbox/
###Don't delete, since it may tell Amazon to download more .sdr
- /documents/*.partial

#SYSTEM#
+ /system/
+ /system/thumbnails/
+ /system/thumbnails/*
- /system/*

#JAILBREAK#
- /acxe/
- /developer/
- /extensions/
- /linkss/
- /python/
- /usbnet/

Attached is an archive of the actual scripts, which you can unpack to your user profile.

[nynaevelan]

Originally Posted by eschwartz

Just thought I'd share a script I use to backup my Kindle sidecar data.

I use a custom column in calibre to make Amazon-sourced books match up to the .sdr data, so I can keep my notes/annotations even for sideloaded books and don't have to worry about syncing to Amazon etc.

The custom column is named "#file", titled "Original Filename". In it, I store the filename that the Amazon book had when I added it from my Kindle, minus the file extension. It will be in the form of {book-title}_{ASIN}.

I use a custom template for sending to device, (limited to the kindle2 device driver, since other devices don't need this,) which looks like this:

program: 
ifempty(
	field('#file'),
	template(
		'{author_sort:sublist(0,1,&)}/{title} - {authors:sublist(0,1,&)}'
	)
)

It uses the contents of custom column "file" as the savepath, if available. If not, (for instance, it isn't originally from Amazon, so I left it blank) it falls back on the main template, which I actually tweaked to only use the first author.

Eschwartz: I am confused, where am I adding this template to use to send to my kindle for the original file name??

[eschwartz]

The device-specific driver. When a Kindle is plugged in, it's in the dropdown under "Device".

Or you can go to Preferences ==> Advanced ==> Plugins ==> Search for plugin (text input box) and search for "kindle".

See screenshots.

[nynaevelan]

Is there a way to tell it to send to a folder in the documents folder that is the same as the tags field??

[eschwartz]

Yes, you can put whatever you want into

template(
	'{your} - {template}/{here}'
)

as the template to use when "#file" is empty.

[nynaevelan]

Originally Posted by eschwartz

Yes, you can put whatever you want into

template(
	'{your} - {template}/{here}'
)

as the template to use when "#file" is empty.

No, not when it is empty, I want it to put in like \{tags}\template program so that it is in the separate folder with the file named from your program.

[eschwartz]

The whole point is to prevent that from happening. It should always overwrite the previous location, to match up to the .sdr folder even after metadata is changed. Otherwise, I'd just stick the books in at the default location.

Wherever the books are, Amazon should still sync annotations to it, when next connected to WiFi. I am striving for semi-independence from Amazon's servers here.

[nynaevelan]

Originally Posted by eschwartz

The whole point is to prevent that from happening. It should always overwrite the previous location, to match up to the .sdr folder even after metadata is changed. Otherwise, I'd just stick the books in at the default location.

Wherever the books are, Amazon should still sync annotations to it, when next connected to WiFi. I am striving for semi-independence from Amazon's servers here.

Ok that makes sense, thanks.

[henriquemaia]

Sorry for necrobumping, but I wanted to thank you for this script. I was wondering if there was a way to backup the sdr files and folders, and found this thread through the search engine.

Thank you for providing such elegant solution. It makes our kindle usage even more reliable.