Fresh daily papers daily with Calibre, on a headless server

[Fastolfe]

Hello everybody,

A few folks asked me to share how I set up my server to deliver fresh newspapers every morning with Calibre, but without running the Calibre frontend, on my server. While Calibre is open-source and documented, the interesting bit (ebook-convert) is surprisingly finicky, buggy and cryptic to use from the command line, so I figured I'd share my setup with you since it works well for me.

In my setup, I have a Linux server, and a fake user called "newspapers". A "newspapers" directory in that fake user's home directory is exported as a public read-only share with Samba under the share name "newspapers". The following script is saved in /home/newspapers/download_newspapers.sh (modify it to suit your needs):

#!/bin/sh
CALIBRE_DIR=/opt/calibre
RECIPES_DIR=$CALIBRE_DIR/resources/recipes
OUTPUT_DIR=/home/newspapers/newspapers
OUTPUT_PROFILE=sony
OUTPUT_FILEEXT=epub
DATE=`date +%d.%m.%Y`

unset DISPLAY
echo $DATE
echo "----------"

# Function that retrieves newspapers
function get_newspaper
{
  OUTDIR=$OUTPUT_DIR/$2
  OUTFILE=$OUTDIR/$3-$DATE.$OUTPUT_FILEEXT
  mkdir -p $OUTDIR
  cd /tmp
  if [ ! -f $OUTFILE ];then
    echo "*** REMOVING OLD ISSUES OF $3... ***"
    rm -f $OUTDIR/$3-??.??.????.$OUTPUT_FILEEXT
    echo "*** DOWNLOADING LATEST $3... ***"
    $CALIBRE_DIR/ebook-convert $RECIPES_DIR/$1 $OUTFILE --output-profile $OUTPUT_PROFILE
  else
    echo "*** NOT DOWNLOADING LATEST $3: ALREADY DONE... ***"
  fi
}

#Get newspapers
get_newspaper american_thinker.recipe		english		american_thinker
get_newspaper bbc.recipe					english		bbc_news
get_newspaper cnetnews.recipe				english		cnet_news
get_newspaper coding_horror.recipe			english		coding_horror
get_newspaper comics_com.recipe				english		comics_com
get_newspaper xkcd.recipe					english		xkcd
get_newspaper dilbert.recipe				english		dilbert
get_newspaper discover_magazine.recipe		english		discover_mag
get_newspaper endgadget.recipe				english		engadget
get_newspaper linux_magazine.recipe			english		linux_mag
get_newspaper linuxdevices.recipe			english		linux_devices
get_newspaper the_new_republic.recipe		english		new_republic
get_newspaper nypost.recipe					english		new_york_post
get_newspaper theonion.recipe				english		the_onion
get_newspaper popscience.recipe				english		popular_science
get_newspaper scientific_american.recipe		english		scientific_am
get_newspaper think_progress.recipe			english		think_progress
get_newspaper wash_post.recipe				english		washington_post
get_newspaper wapo_cartoons.recipe			english		wa_post_cartoons
get_newspaper wikinews_en.recipe			english		wikinews
get_newspaper wired_daily.recipe			english		wired_mag

get_newspaper elmundo.recipe				espanol		el_mundo
get_newspaper el_pais.recipe				espanol		el_pais

get_newspaper dhnet_be.recipe				francais		dh
get_newspaper lameuse_be.recipe				francais		la_meuse
get_newspaper lesoir_be.recipe				francais		le_soir

and then this line in the "newspapers" user's crontab:

0 3 * * *		$HOME/download_newspapers.sh 2>&1> $HOME/download_newspapers.log

Then, for good measure, disable login for user "newspapers" by replacing the shell (/bin/bash usually) with /bin/false in /etc/passwd.

This setup will begin downloading all the newspapers mentioned in the script and save them in /home/newspapers/newspapers/<language>/<name>.epub in the special Sony "periodicals" EPUB format, everyday at 3am. If you need another format, change the extension of the output file and the OUTPUT_PROFILE variable accordingly in the script.

After that, it's trivial to retrieve those epubs automatically. For example:

- For myself, I usually plug my Sony Reader's memory stick card in a card reader connected to the server in the evening, and at 5am, the server automatically removes old issues of the newspapers in the card and replaces them with the fresh ones that were downloaded during the night. For that, the MS is automatically mounted under /mnt/mscard when I plug it in (depends on your distro and the volume name you gave your card) and I have this line in my crontab:

0 5 * * *		rsync -avx --delete /home/newspapers/newspapers/ /mnt/mscard/newspapers/

- For my Windows users, they connect the \\<server name>\newspapers network drive and plug in their ebook reader. To ensure the network drive and the ebook reader drive always get the same drive letters assigned, they use USBDLM. Then, they use Syncback to synchronize the content of the "newspapers" network drive (or a subset of it, if they don't want all the languages) and whatever destination directory they chose in their ebook reader. Syncback can even disconnect the ebook reader's drive letter automatically, so all they need to do to load up the newspapers in the device in the morning is plug it, double-click the Syncback shortcut, wait for it to complete and unplug the device safely.

One remark about picking which newspapers to download:

if you add a newspaper to the list in the script, try that new download alone from the command line first and watch carefully the output of ebook-convert. Some newspapers's webpages will get downloaded, but the ebook generation process will fail (for example, the french newspaper "Le Monde", which seems to fail because of invalid characters), some will be so slow to download it'll take 2 hours, and some will send the ebook-convert process into an infinite loop that'll burn CPU and hang the entire process forever (for example, the Huffington Post). So in short, ensure the newspaper can be downloaded in a reasonable amount of time and generates an epub without hanging the process before including it into the live script.

Also, don't forget to update your copy of Calibre as soon as a new version comes out, as the news download recipes are updated regularly. I suspect at some point they really should start distributing those recipes as a separate package incidentally.

So, there's my solution to automatically get fresh newspapers daily. My users and I like it, and it sure beats leaving a copy of Calibre running in an open X session (something my server doesn't have anyway). I hope it'll help you get going if you want the same setup. If your ebook has WiFi, I bet you can make it even more hassle-free than what we got going here.

Have fun!

[Doogman]

I was planning on looking into this function of Calibre for my own headless home server. You saved me a bunch of time with this script.

[Pelesz]

That's almost what i need, thank you very much.