The Guardian 24 automatic download

[carandol]

Originally Posted by daudi

Can you look at the CF in a PC? Is it possible that the PDF has been downloaded okay and it in the subdirectory, but because there is no manifest file it won't open?

The manifest file is there, correctly showing when download (supposedly) took place. But the Guardian.pdf is a 0B Empty Document.

[Alcuith]

Originally Posted by carandol

The manifest file is there, correctly showing when download (supposedly) took place. But the Guardian.pdf is a 0B Empty Document.

I'm getting exactly the same thing. Icon shows in the Iliad, but blinks and no content.

Just had a look at the card in the PC, manifest file is there but the pdf is 0kb

[cen]

Made a bit of progress. Looks like the iLiad has a problem writing to my CF card (Transcend 1GB 80x). Changing to internal media (/mnt/free) and the program works fine.

Alcuith are you using CF or other storage? It would be interesting to see if this works for you and carandol if you use internal storage or mmc as rio uses.

[Falzon]

I had a similar problem running the script - that executed and copied files to CF (PNY OPTIMA 4GB 60x) via wireless.sh start <SSID> wpa <WPA_KEY>
It was going through the motions (flashing green light, folder, manifest and icon created but pdf with 0kb size.)
I used mrxvt to view the executing script and noticed it would STALL during the transfer (varying between 7% and 58% complete) then would time out thus producing the null pdf.

I took the advice of CEN (thanks) and changed the path to the MCC (/media/card/) and it works every time.
The whole transfer takes less then 10 seconds, as opposed to the minute or so it previously took before it would timeout.

I haven't yet tried it with internal memory (/mnt/free/) but guess this would also work.

Can we infer that this may be an issue with wireless transfer of files directly to CF? Probably due to the limited max transfer rate of the CF not keeping up with the wireless transfer rate.
Could faster CF cards work any better I wonder?

[carandol]

Yay! Success at last! I moved the program to internal memory, added the connection details to the wireless command, and it works perfectly. It must be an issue with the speed of CF cards, as Falzon suggests. one and all!

[carandol]

Now that I've got this working, I've tinkered with the script a bit, so that instead of just downloading the World News, it downloads all the sections of G24; Top Stories, World News, Business News, Media News and Sports News. If you don't want any of the sections, remove (or hash out) the relevant bit of script.

I've included a few tips in comments in the script for setting it up, with alternate lines for internal memory and MMC card (though not for CF card because it doesn't seem to work).

Anyway, here's the code, and the whole thing's zipped below.

#!/bin/sh
export GUARDIAN_PATH=/mnt/free/newspapers/Guardian24 ## for internal memory
#export GUARDIAN_PATH=/media/card/newspapers/Guardian24 ## for MMC card
### Note: doesn't seem to work with CF card
### Connect -- insert details for wireless network
/usr/bin/wired.sh start 
/usr/bin/wireless.sh start <SSID> <ENC_TYPE> <PASS>
### Top Stories ###
rm -rf $GUARDIAN_PATH/Topstories.pdf
mkdir $GUARDIAN_PATH/Topstories.pdf
echo "<?xml version=\"1.0\" encoding=\"utf-8\"?><package><metadata><dc-metadata><Title>The Guardian 24h Top Stories</Title><Description>Downloaded: `echo \`date\``</Description></dc-metadata><y-metadata><startpage>Topstories.pdf</startpage><image>icon.png</image><version>000</version></y-metadata></metadata></package>" > $GUARDIAN_PATH/Topstories.pdf/manifest.xml
wget -O $GUARDIAN_PATH/Topstories.pdf/Topstories.pdf http://download.guardian.co.uk/g24/Topstories.pdf
cp $GUARDIAN_PATH/get-guardian/icon.png $GUARDIAN_PATH/Topstories.pdf/icon.png
### World News ###
rm -rf $GUARDIAN_PATH/World.pdf
mkdir $GUARDIAN_PATH/World.pdf
echo "<?xml version=\"1.0\" encoding=\"utf-8\"?><package><metadata><dc-metadata><Title>The Guardian 24h World News</Title><Description>Downloaded: `echo \`date\``</Description></dc-metadata><y-metadata><startpage>World.pdf</startpage><image>icon.png</image><version>000</version></y-metadata></metadata></package>" > $GUARDIAN_PATH/World.pdf/manifest.xml
wget -O $GUARDIAN_PATH/World.pdf/World.pdf http://download.guardian.co.uk/g24/World.pdf
cp $GUARDIAN_PATH/get-guardian/icon.png $GUARDIAN_PATH/World.pdf/icon.png
### Business ###
rm -rf $GUARDIAN_PATH/Business.pdf
mkdir $GUARDIAN_PATH/Business.pdf
echo "<?xml version=\"1.0\" encoding=\"utf-8\"?><package><metadata><dc-metadata><Title>The Guardian 24h Business News</Title><Description>Downloaded: `echo \`date\``</Description></dc-metadata><y-metadata><startpage>Business.pdf</startpage><image>icon.png</image><version>000</version></y-metadata></metadata></package>" > $GUARDIAN_PATH/Business.pdf/manifest.xml
wget -O $GUARDIAN_PATH/Business.pdf/Business.pdf http://download.guardian.co.uk/g24/Business.pdf
cp $GUARDIAN_PATH/get-guardian/icon.png $GUARDIAN_PATH/Business.pdf/icon.png
### Media ###
rm -rf $GUARDIAN_PATH/Media.pdf
mkdir $GUARDIAN_PATH/Media.pdf
echo "<?xml version=\"1.0\" encoding=\"utf-8\"?><package><metadata><dc-metadata><Title>The Guardian 24h Media News</Title><Description>Downloaded: `echo \`date\``</Description></dc-metadata><y-metadata><startpage>Media.pdf</startpage><image>icon.png</image><version>000</version></y-metadata></metadata></package>" > $GUARDIAN_PATH/Media.pdf/manifest.xml
wget -O $GUARDIAN_PATH/Media.pdf/Media.pdf http://download.guardian.co.uk/g24/Media.pdf
cp $GUARDIAN_PATH/get-guardian/icon.png $GUARDIAN_PATH/Media.pdf/icon.png
### Sport ###
rm -rf $GUARDIAN_PATH/Sport.pdf
mkdir $GUARDIAN_PATH/Sport.pdf
echo "<?xml version=\"1.0\" encoding=\"utf-8\"?><package><metadata><dc-metadata><Title>The Guardian 24h Sport News</Title><Description>Downloaded: `echo \`date\``</Description></dc-metadata><y-metadata><startpage>Sport.pdf</startpage><image>icon.png</image><version>000</version></y-metadata></metadata></package>" > $GUARDIAN_PATH/Sport.pdf/manifest.xml
wget -O $GUARDIAN_PATH/Sport.pdf/Sport.pdf http://download.guardian.co.uk/g24/Sport.pdf
cp $GUARDIAN_PATH/get-guardian/icon.png $GUARDIAN_PATH/Sport.pdf/icon.png
### Disconnect
/usr/bin/wired.sh stop
/usr/bin/wireless.sh stop

[-Thomas-]

Originally Posted by Falzon

Can we infer that this may be an issue with wireless transfer of files directly to CF? Probably due to the limited max transfer rate of the CF not keeping up with the wireless transfer rate.
Could faster CF cards work any better I wonder?

There is a known bug on the iLiad when using WiFi and the CF card at the same time. The transfer rate goes down to ~1kb/s in this case. This isn't specific to any CF brand or model, so exchanging the CF card won't help either.

I think you could enhance the script to download the file to /tmp first and move it to the CF card afterwards...

[cen]

Originally Posted by carandol

I've included a few tips in comments in the script for setting it up, with alternate lines for internal memory and MMC card (though not for CF card because it doesn't seem to work).

Here is a version with working CF. The script will download news to <media of choice>/newspapers/Guardian24/. All files are downloaded to /tmp on internal memory first then moved to the correct media.

Things to do before running script
1) Uncomment the line for your media of choice
2) If using wifi set the ssid / enc / key

#!/bin/sh

####
# Uncomment one of the following depending on storage device
#export GUARDIAN_PATH=/media/cf/newspapers/Guardian24   # CF storage
#export GUARDIAN_PATH=/media/card/newspapers/Guardian24 # SD/MMC card
#export GUARDIAN_PATH=/mnt/free/newspapers/Guardian24   # Internal storage


/usr/bin/wired.sh start 
/usr/bin/wireless.sh start <SSID> <ENC_TYPE> <KEY>

for i in World Business Media Sport Topstories
do
	rm -rf $GUARDIAN_PATH/$i.pdf
	mkdir -p $GUARDIAN_PATH/$i.pdf

	echo "<?xml version=\"1.0\" encoding=\"utf-8\"?><package><metadata><dc-metadata><Title>The Guardian 24h - $i</Title><Description>Downloaded: `echo \`date\``</Description></dc-metadata><y-metadata><startpage>$i.pdf</startpage><image>icon.png</image><version>000</version></y-metadata></metadata></package>" > $GUARDIAN_PATH/$i.pdf/manifest.xml

	wget -O /tmp/$i.pdf http://download.guardian.co.uk/g24/$i.pdf
	mv /tmp/$i.pdf $GUARDIAN_PATH/$i.pdf/$i.pdf

	cp $GUARDIAN_PATH/get-guardian/icon.png $GUARDIAN_PATH/$i.pdf/icon.png
done

/usr/bin/wired.sh stop
/usr/bin/wireless.sh stop

[carandol]

Thanks cen! I knew there had to be a more elegant way of doing that, but I've not done any scripting since I stopped using DOS in the early '90s, so I was pleased I'd got it working at all

[DiaLogical]

Hi guys, I recently got an iLiad myself and it's great to see all kind of user created content. And this one in particular has me very interested. Since I didn't have a problem with other 3th party progs like the browser, reader(s), etc and the code seems clean I didn't expect any now either. But for some reason it doesn't seem to actually create or rather get the required .pdf file.

In short this is all I did;

• extracted the latest .zip folder on my iLiad's internal memory in the newspaper folder (icon, manifest and run file)
• changed the following to the run file (see quote below; uncomment + wireless info)
• Selected the Guardian UK run file on the iLiad, it gives the blinking loading bar, after say 10 seconds the screen renews and nothing happened or opened. No .pdf file (not even a 0 kb) either.

The information about my wireless network is definitely correct, that's for the wep key and the SSID name, the ENC_TYPE is 128bit wep so I filled in <wep> lowercase, which should be fine as well I guess.

Any idea what could be the problem here? Thanks in advance.

#!/bin/sh

####
# Uncomment one of the following depending on storage device
#export GUARDIAN_PATH=/media/cf/newspapers/Guardian24 # CF storage
#export GUARDIAN_PATH=/media/card/newspapers/Guardian24 # SD/MMC card
export GUARDIAN_PATH=/mnt/free/newspapers/Guardian24 # Internal storage


/usr/bin/wired.sh start
/usr/bin/wireless.sh start <mynetworkSSIDname> <wep> <WEP0EX1AM2PL3E>

for i in World Business Media Sport Topstories
do
rm -rf $GUARDIAN_PATH/$i.pdf
mkdir -p $GUARDIAN_PATH/$i.pdf

echo "<?xml version=\"1.0\" encoding=\"utf-8\"?><package><metadata><dc-metadata><Title>The Guardian 24h - $i</Title><Description>Downloaded: `echo \`date\``</Description></dc-metadata><y-metadata><startpage>$i.pdf</startpage><image>icon.png</image><version>000</version></y-metadata></metadata></package>" > $GUARDIAN_PATH/$i.pdf/manifest.xml

wget -O /tmp/$i.pdf http://download.guardian.co.uk/g24/$i.pdf
mv /tmp/$i.pdf $GUARDIAN_PATH/$i.pdf/$i.pdf

cp $GUARDIAN_PATH/get-guardian/icon.png $GUARDIAN_PATH/$i.pdf/icon.png
done

/usr/bin/wired.sh stop
/usr/bin/wireless.sh stop

[cen]

replace:

/usr/bin/wireless.sh start <mynetworkSSIDname> <wep> <WEP0EX1AM2PL3E>

with

/usr/bin/wireless.sh start mynetworkSSIDname wep WEP0EX1AM2PL3E

Probably should have made it clearer that the everything between and including the < > need to be replaced.

[DiaLogical]

That did the trick indeed, and now it sounds like a stupid mistake)
Thanks a bunch, appreciate the quick reply!

One more question, the font of the Guardian pdf files seems to be rather on the small side. Is there another quick way to enlarge them a few points instead of manually zooming with the iLiad's build in function every time?

[carandol]

Originally Posted by DiaLogical

That did the trick indeed, and now it sounds like a stupid mistake)
Thanks a bunch, appreciate the quick reply!

Well, I looked at your code and couldn't see what was wrong with it either, until cen pointed it out

Originally Posted by DiaLogical

One more question, the font of the Guardian pdf files seems to be rather on the small side. Is there another quick way to enlarge them a few points instead of manually zooming with the iLiad's build in function every time?

I think the short answer to that is no. The limitation of PDF files is that they're meant to be displayed exactly as the layout was designed. You can zoom in, but you can't alter the font size. It's either zoom in or buy a magnifying glass

[DiaLogical]

Originally Posted by carandol

I think the short answer to that is no. The limitation of PDF files is that they're meant to be displayed exactly as the layout was designed. You can zoom in, but you can't alter the font size. It's either zoom in or buy a magnifying glass

In that case I think I'll just go for the first option)

If anyone would know of other newspapers who publish a daily pdf version as well, I'd be glad to hear about it. Most newssites have 'mobile' versions, but so far I haven't seen any others organised on one page/in one file.

Edit: Did some more thorough searching and found a few, e.g. wikinews. But no luck on the 'big ones' yet as BBC, CNN,...
Edit2: Wall street journal. It seems feedbooks has more of these good news RSS feeds which (they) convert to pdf. Would it be possible to change the script as to download the respective pdf files from there (e.g. the Wall street journal, BBC news) as well?

[cen]

Originally Posted by DiaLogical

But no luck on the 'big ones' yet as BBC, CNN,...

There is a scraped html version of the BBC website available.

It seems feedbooks has more of these good news RSS feeds which (they) convert to pdf.

Take a look at FeedBooks iNewsStand