Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book General > General Discussions

Notices

Reply
 
Thread Tools Search this Thread
Old 12-19-2010, 04:53 PM   #1
Fastolfe
Bookworm
Fastolfe makes transoceanic flights without the assistance of an airplaneFastolfe makes transoceanic flights without the assistance of an airplaneFastolfe makes transoceanic flights without the assistance of an airplaneFastolfe makes transoceanic flights without the assistance of an airplaneFastolfe makes transoceanic flights without the assistance of an airplaneFastolfe makes transoceanic flights without the assistance of an airplaneFastolfe makes transoceanic flights without the assistance of an airplaneFastolfe makes transoceanic flights without the assistance of an airplaneFastolfe makes transoceanic flights without the assistance of an airplaneFastolfe makes transoceanic flights without the assistance of an airplaneFastolfe makes transoceanic flights without the assistance of an airplane
 
Fastolfe's Avatar
 
Posts: 98
Karma: 55796
Join Date: Dec 2010
Device: Sony PRS-650
Fresh daily papers daily with Calibre, on a headless server

Hello everybody,

A few folks asked me to share how I set up my server to deliver fresh newspapers every morning with Calibre, but without running the Calibre frontend, on my server. While Calibre is open-source and documented, the interesting bit (ebook-convert) is surprisingly finicky, buggy and cryptic to use from the command line, so I figured I'd share my setup with you since it works well for me.

In my setup, I have a Linux server, and a fake user called "newspapers". A "newspapers" directory in that fake user's home directory is exported as a public read-only share with Samba under the share name "newspapers". The following script is saved in /home/newspapers/download_newspapers.sh (modify it to suit your needs):

Code:
#!/bin/sh
CALIBRE_DIR=/opt/calibre
RECIPES_DIR=$CALIBRE_DIR/resources/recipes
OUTPUT_DIR=/home/newspapers/newspapers
OUTPUT_PROFILE=sony
OUTPUT_FILEEXT=epub
DATE=`date +%d.%m.%Y`

unset DISPLAY
echo $DATE
echo "----------"

# Function that retrieves newspapers
function get_newspaper
{
  OUTDIR=$OUTPUT_DIR/$2
  OUTFILE=$OUTDIR/$3-$DATE.$OUTPUT_FILEEXT
  mkdir -p $OUTDIR
  cd /tmp
  if [ ! -f $OUTFILE ];then
    echo "*** REMOVING OLD ISSUES OF $3... ***"
    rm -f $OUTDIR/$3-??.??.????.$OUTPUT_FILEEXT
    echo "*** DOWNLOADING LATEST $3... ***"
    $CALIBRE_DIR/ebook-convert $RECIPES_DIR/$1 $OUTFILE --output-profile $OUTPUT_PROFILE
  else
    echo "*** NOT DOWNLOADING LATEST $3: ALREADY DONE... ***"
  fi
}

#Get newspapers
get_newspaper american_thinker.recipe		english		american_thinker
get_newspaper bbc.recipe					english		bbc_news
get_newspaper cnetnews.recipe				english		cnet_news
get_newspaper coding_horror.recipe			english		coding_horror
get_newspaper comics_com.recipe				english		comics_com
get_newspaper xkcd.recipe					english		xkcd
get_newspaper dilbert.recipe				english		dilbert
get_newspaper discover_magazine.recipe		english		discover_mag
get_newspaper endgadget.recipe				english		engadget
get_newspaper linux_magazine.recipe			english		linux_mag
get_newspaper linuxdevices.recipe			english		linux_devices
get_newspaper the_new_republic.recipe		english		new_republic
get_newspaper nypost.recipe					english		new_york_post
get_newspaper theonion.recipe				english		the_onion
get_newspaper popscience.recipe				english		popular_science
get_newspaper scientific_american.recipe		english		scientific_am
get_newspaper think_progress.recipe			english		think_progress
get_newspaper wash_post.recipe				english		washington_post
get_newspaper wapo_cartoons.recipe			english		wa_post_cartoons
get_newspaper wikinews_en.recipe			english		wikinews
get_newspaper wired_daily.recipe			english		wired_mag

get_newspaper elmundo.recipe				espanol		el_mundo
get_newspaper el_pais.recipe				espanol		el_pais

get_newspaper dhnet_be.recipe				francais		dh
get_newspaper lameuse_be.recipe				francais		la_meuse
get_newspaper lesoir_be.recipe				francais		le_soir
and then this line in the "newspapers" user's crontab:

Code:
0 3 * * *		$HOME/download_newspapers.sh 2>&1> $HOME/download_newspapers.log
Then, for good measure, disable login for user "newspapers" by replacing the shell (/bin/bash usually) with /bin/false in /etc/passwd.

This setup will begin downloading all the newspapers mentioned in the script and save them in /home/newspapers/newspapers/<language>/<name>.epub in the special Sony "periodicals" EPUB format, everyday at 3am. If you need another format, change the extension of the output file and the OUTPUT_PROFILE variable accordingly in the script.

After that, it's trivial to retrieve those epubs automatically. For example:

- For myself, I usually plug my Sony Reader's memory stick card in a card reader connected to the server in the evening, and at 5am, the server automatically removes old issues of the newspapers in the card and replaces them with the fresh ones that were downloaded during the night. For that, the MS is automatically mounted under /mnt/mscard when I plug it in (depends on your distro and the volume name you gave your card) and I have this line in my crontab:

Code:
0 5 * * *		rsync -avx --delete /home/newspapers/newspapers/ /mnt/mscard/newspapers/
- For my Windows users, they connect the \\<server name>\newspapers network drive and plug in their ebook reader. To ensure the network drive and the ebook reader drive always get the same drive letters assigned, they use USBDLM. Then, they use Syncback to synchronize the content of the "newspapers" network drive (or a subset of it, if they don't want all the languages) and whatever destination directory they chose in their ebook reader. Syncback can even disconnect the ebook reader's drive letter automatically, so all they need to do to load up the newspapers in the device in the morning is plug it, double-click the Syncback shortcut, wait for it to complete and unplug the device safely.

One remark about picking which newspapers to download:

if you add a newspaper to the list in the script, try that new download alone from the command line first and watch carefully the output of ebook-convert. Some newspapers's webpages will get downloaded, but the ebook generation process will fail (for example, the french newspaper "Le Monde", which seems to fail because of invalid characters), some will be so slow to download it'll take 2 hours, and some will send the ebook-convert process into an infinite loop that'll burn CPU and hang the entire process forever (for example, the Huffington Post). So in short, ensure the newspaper can be downloaded in a reasonable amount of time and generates an epub without hanging the process before including it into the live script.

Also, don't forget to update your copy of Calibre as soon as a new version comes out, as the news download recipes are updated regularly. I suspect at some point they really should start distributing those recipes as a separate package incidentally.

So, there's my solution to automatically get fresh newspapers daily. My users and I like it, and it sure beats leaving a copy of Calibre running in an open X session (something my server doesn't have anyway). I hope it'll help you get going if you want the same setup. If your ebook has WiFi, I bet you can make it even more hassle-free than what we got going here.

Have fun!

Last edited by Fastolfe; 12-20-2010 at 05:17 AM.
Fastolfe is offline   Reply With Quote
Old 12-19-2010, 08:03 PM   #2
Doogman
Connoisseur
Doogman has learned how to read e-booksDoogman has learned how to read e-booksDoogman has learned how to read e-booksDoogman has learned how to read e-booksDoogman has learned how to read e-booksDoogman has learned how to read e-booksDoogman has learned how to read e-booksDoogman has learned how to read e-books
 
Doogman's Avatar
 
Posts: 52
Karma: 945
Join Date: Jun 2010
Device: Nook ST


I was planning on looking into this function of Calibre for my own headless home server. You saved me a bunch of time with this script.

Doogman is offline   Reply With Quote
 
Enthusiast
Old 12-20-2010, 05:29 AM   #3
Pelesz
Member
Pelesz began at the beginning.
 
Pelesz's Avatar
 
Posts: 11
Karma: 26
Join Date: Dec 2009
Location: Transdanubia Meridianus
Device: Pocketbook 515, Hanvon n516 (OI)
Thumbs up

That's almost what i need, thank you very much.
Pelesz is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[Old Thread] News auto download and email on headless server mcphail Related Tools 9 11-19-2011 03:50 AM
Calibre recipe for daily Portuguese newspaper "Correio da Manhã" jmst Recipes 2 11-01-2010 01:01 PM
Headless Calibre Server Setup godzilla8nj Related Tools 4 03-10-2010 01:39 PM
Calibre not detecting Sony Reader Daily (PRS-900) on Vista adikira Calibre 15 03-04-2010 04:54 PM
The daily commute - books vs ebook readers vs papers ShortNCuddlyAm Lounge 33 02-10-2010 09:14 PM


All times are GMT -4. The time now is 09:23 AM.


MobileRead.com is a privately owned, operated and funded community.