Script to fetch & send to Kindle (Mac/Linux)

[clanger9]

Hi all,

After having problems with Calibre intermittently failing to fetch and email news to my Kindle, I wrote a script to automate the process.

You can easily schedule the script on a Mac (with launchd) or Linux (with cron). It is a bit rough and ready but seems to work OK. The main feature is that it will retry fetching and sending a number of times. It will also notice if either of these processes gets stuck/hung and kill and restart the process after a specified time.

It is designed to work with a Kindle and a Gmail account, but it should be adaptable to work with anything that Calibre supports. You have to save your Gmail password in the file. This is horribly insecure, so please don't use your main email account (except for testing). Far better to create a dummy Gmail account for sending to the Kindle.

You will need to change KindleMail, GoogleUser and GooglePassword as a minimum. If you are on anything other than a Mac, you'll need to change BinDir and RecipeDir as well.

Have fun!

#!/bin/bash

### CHANGE THESE VALUES ###
KindleMail='<your kindle email>@free.kindle.com'
GoogleUser='<your gmail account>@gmail.com'
GooglePassword='<your gmail password>'
### CHANGE THESE VALUES ###

BinDir='/Applications/calibre.app/Contents/MacOS'
RecipeDir='/Applications/calibre.app/Contents/Resources/resources/recipes'

FetchTimeout=7200  # Allow up to 120 minutes to fetch news 
SendTimeout=300    # Allow up to 5 minutes to send email
MaxIterations=3

TempFile="/tmp/$1.mobi"
CmdFetch="$BinDir/ebook-convert $RecipeDir/$1.recipe $TempFile --output-profile kindle $2 $3"
CmdSend="$BinDir/calibre-smtp -r smtp.gmail.com --port 587 --username $GoogleUser --password $GooglePassword --a $TempFile $GoogleUser $KindleMail $1"

PN=`basename "$0"`
VER='1.07'

# Usage instructions
if [ ! $1 ]; then
	echo "error: you must specify a recipe file in $RecipeDir" 1>&2
	exit 1
elif [ ! -f $RecipeDir/$1.recipe ]; then
	echo "error: recipe file $RecipeDir/$1.recipe not found" 1>&2
	exit 1
fi

# Start fetch process
ErrorFlag=0
i=1
while [ $i -le $MaxIterations ]
do
	# Wait a number of seconds before retrying
	if [ $ErrorFlag -eq 1 ]; then
		PauseTime=$(expr 30 \* $i)
		echo "`date` $PN warning: failed to fetch $1, making attempt $i of $MaxIterations in $PauseTime seconds" 1>&2
		sleep $PauseTime
	fi
	
	# Start fetch task in the background and get its process id
	nice $CmdFetch >/dev/null 0<&1 2>&1 &
	TaskID1=$!

	# Set watchdog timer to kill fetch task after specified time
	( 
		sleep $FetchTimeout 
		kill $TaskID1 >/dev/null 0<&1 2>&1
	) >/dev/null 0<&1 2>&1 & 
	WatchdogID1=$!

	# Bring the fetch task back to the foreground
	trap "kill $TaskID1 $WatchdogID1 >/dev/null 0<&1 2>&1; exit" INT TERM EXIT
	wait $TaskID1
	TaskSuccess=$?
	trap - INT TERM EXIT
	kill $WatchdogID1 >/dev/null 0<&1 2>&1
	
	# Exit loop if task completed successfully
	if [ $TaskSuccess -eq 0 ]; then
		ErrorFlag=0
		break
	else
		ErrorFlag=1
		i=$(($i+1))
		continue
	fi
done

# If fetching failed, print error and quit
if [ $ErrorFlag -eq 1 ]; then
	rm $TempFile >/dev/null 0<&1 2>&1
	echo "`date` $PN error: failed to fetch $1, giving up" 1>&2
	exit 1
fi

# Start sending process

ErrorFlag=0
i=1
while [ $i -le $MaxIterations ]
do
	# Wait a number of seconds before retrying
    if [ $ErrorFlag -eq 1 ]; then
		PauseTime=$(expr 30 \* $i)
		echo "`date` $PN warning: failed to send $1, making attempt $i of $MaxIterations in $PauseTime seconds" 1>&2
		sleep $PauseTime
	fi
	
	# Start send task in the background and get its process id
	nice $CmdSend >/dev/null 0<&1 2>&1 &
	TaskID2=$!
	# Set watchdog timer to kill send task after specified time
	(
		sleep $SendTimeout
		kill $TaskID2 >/dev/null 0<&1 2>&1
	) >/dev/null 0<&1 2>&1 &
	WatchdogID2=$!

	# Bring the send task back to the foreground
	trap "kill $TaskID2 $WatchdogID2 >/dev/null 0<&1 2>&1; exit" INT TERM EXIT
	wait $TaskID2
	TaskSuccess=$?
	trap - INT TERM EXIT
	kill $WatchdogID2 >/dev/null 0<&1 2>&1
	
	# Exit loop if task completed successfully
	if [ $TaskSuccess -eq 0 ]; then
		ErrorFlag=0
		break
	else
		ErrorFlag=1
		i=$(($i+1))
		continue
	fi
done

# Clean up
kill $(jobs -pr) >/dev/null 0<&1 2>&1
rm $TempFile >/dev/null 0<&1 2>&1

# If sending failed, print error and quit
if [ $ErrorFlag -eq 1 ]; then
	echo "`date` $PN error: failed to send $1 to $KindleMail, giving up" 1>&2
	exit 2
else
	echo "`date` $PN sent $1 to $KindleMail"
	exit 0
fi

[dennisMe]

I'm just tinkering with this script now and thought I'd sign up here to give you a big thank you!
Just what the doctor ordered!

Dennis

[clanger9]

Hi Dennis,


Glad to hear the script is of use!

It's been working pretty reliably for me here, although I think there is a small error somewhere because it doesn't always re-try the fetch for some reason. This happens so infrequently I haven't really looked into it, so if you spot any mistakes please let me know.

In case anyone is using this script on a Mac, here's how to set the script to run automatically:

1. Save the script above to ~/Library/Scripts/calibre_fetch.sh, having changed the values highlighted in red.
2. Set the correct permissions with

chmod 700 ~/Library/Scripts/calibre_fetch.sh

Now you have a generic fetch script. Next, create a set of XML launchd schedules. These live in ~/Library/LaunchAgents.

The following example will fetch Dilbert each day for user "Paul":

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>Debug</key>
	<false/>
	<key>Label</key>
	<string>com.calibre.dilbert</string>
	<key>LowPriorityIO</key>
	<true/>
	<key>ProgramArguments</key>
	<array>
		<string>/Users/Paul/Library/Scripts/calibre_fetch.sh</string>
		<string>dilbert</string>
	</array>
	<key>StartCalendarInterval</key>
	<dict>
		<key>Hour</key>
		<integer>7</integer>
		<key>Minute</key>
		<integer>10</integer>
	</dict>
</dict>
</plist>

Save the file to ~/Library/LaunchAgents/com.calibre.dilbert.plist

The above example should give you the basic idea (you will need to adjust the "/Users/Paul/" absolute path and change the "dilbert" references to the recipe you want). Unfortunately the launchd plist syntax is fairly arcane and seems to vary with each release of Mac OS. The above is designed to work with Tiger (10.4.x). You might want to use Lingon to create these files, or read the launchd plist documentation.

Once you've created the launch agent, put it to work:
1. Set the correct permissions on ~/Library/LaunchAgents with

chmod 700 ~/Library/LaunchAgents

2. Load the launch agent with

launchctl load ~/Library/LaunchAgents/com.calibre.dilbert.plist

That's it! Mac OS will re-load the launch agent if the machine is rebooted, so from here on it should all 'just work'. You can watch the output from the script in the Mac OS Console if things don't happen as expected.

If you want to kick-start the script (to check it all works), then do so with

launchctl start com.calibre.dilbert

You can add as many launch agents as you want to ~/Library/LaunchAgents.

Have fun!

[dennisMe]

on line 38 the sleep command thows an error (for the record: Open Suse 11.4).

This version works better for me (expr used and * escaped):
sleep $(expr 30 \* $i)


Dennis

[clanger9]

Just a quick update: I finally got to the bottom of why the script doesn't retry properly.
It turns out that while...do...done works as expected (whereas do...loop doesn't), so I changed the script above to use that.

I also incorporated dennisMe's suggestion (thanks for that!).

It still leaves `sleep` processes lying around (until they time out), despite trying to clean up afterwards. I haven't figured out how to kill them properly when the script completes, as the captured PID doesn't get the child process.

If anyone knows how to do this properly, please let me know...

[Mixx]

Hi,

I have trouble locating the recipes folder on Ubuntu Linux, can somebody please tell me where it is hidden?

It looks as if the recipes were stored in the database. If so, how can I use a script like this? I could slightly modify of course the recipes of interest in Calibre, in which case they are saved in ahidden folder in my home directory.

I am tinkering with this script and would like to install it for Ununtu 12.04 to test.

Thanxx, Mixx