Hi folks,

I've created a framework for automatic content downloaders like the Guardian24 (http://www.mobileread.com/forums/showthread.php?t=22319) or the comic strip (http://www.mobileread.com/forums/showthread.php?t=12666) downloader. It allows users to control which content they want to retrieve automatically while connecting to the iDS (yes, that's correct - no connection scripts etc., just hit the button at the upper left :)). Downloaders are handled as plug-ins and can be developed easily.

Attached are two packages: The autodl-v0.2.zip contains an installer for the framework itself, including a dummy plugin. The guardian.zip is a converted Guardian24 downloader to show you how a simple plugin might look like.

Here's how to install the framework:

Unzip the autodl-v0.2.zip to your internal memory, CF card or SD card - it doesn't matter
Install the framework by selecting the newly created item. After the installation you have a new item in the "documents" folder on the internal memory: "Automatic Downloaders". It contains the control scripts for the plugins
To install the Guardian24 downloader, unzip the guardian.zip to /mnt/free/download/.
Connect to the iDS by hitting the iDS hardware button; the iDS software says "No new files" (or something similar) - this is intended behaviour and can be ignored.
Afterwards you have the newest Guardian24 issue in your news directory.


Available plugins:

20minutes (fr), 20minutos (es) (http://www.mobileread.com/forums/showpost.php?p=246213&postcount=27) (thx paulgreg)
Comics (http://www.mobileread.com/forums/showpost.php?p=210233&postcount=4)
Feedbooks RSS subscriptions (http://www.mobileread.com/forums/showpost.php?p=324601&postcount=35)
Fictionwise purchased books (http://www.mobileread.com/forums/showpost.php?p=211105&postcount=6)
Sunday Times Software Development (http://www.mobileread.com/forums/showpost.php?p=255937&postcount=29) (thx paulgreg)
Tagesschau (de) (http://www.mobileread.com/forums/showpost.php?p=211273&postcount=8)
The Guardian (attached)


Here's how to develop your own plug-in:

Copy the /mnt/free/download/dummy/ directory to your PC
Modify the manifest.xml and edit the name of your downloader
Modify the scripts/main.sh, this is the script that will be executed when connecting
Rename the "dummy" directory to something else...
Copy the directory back to /mnt/free/download/... that's it!

It's really awesome to have content downloaded with just one click, so hopefully there will be many plug-ins available in the future :)

EDIT: v0.2: fixed a bug during installation which prevented the script from working ;)

This is a really good idea. Great Job!

WoW! This is excellent :)

Here are two more downloaders: Adam's Daily Comic Strip (http://www.mobileread.com/forums/showthread.php?t=12582) and sinman's 20minutos.es (http://www.mobileread.com/forums/showpost.php?p=166122&postcount=32). Simply extract them to /mnt/free/download/.

Edit: There's an update on the 20minutos script from Gregory, check out his post (http://www.mobileread.com/forums/showpost.php?p=246213&postcount=27) for details.

Without any parsing you could also directly link to feeds or newspaper on Feedbooks. For example: http://www.feedbooks.com/feed/743.mobi will download the RSS feeds for Mobileread in Mobipocket.

I've just completed a very cool script...

The attached plugin allows you to download newly purchased books from Fictionwise.com directly to your device. Simply unzip it into /mnt/free/download/ and add your username/password to the scripts/config.sh file. Works like a charm :D

A plugin for Feedbooks.com RSS subscriptions is coming in the near future... :thumbsup:

Note that this comics script currently goes to /mnt/cf/books and the Guardian24 script goes to /mnt/cf/newspapers -- reasonable choices but had me searching a bit.

Sorry, I forgot to point that out, this can be changed in the scripts/main.sh files. I've found a nice way to automate things: For the attached tagesschau.de plugin I used Mathijs' regutil (http://www.mobileread.com/forums/showthread.php?t=23668) to determine the correct storage type for news...

Please, could someone stop me? I have to write a seminar paper until next tuesday, but this project prevents me from doing so :D

sorry, I'm not going to be the one who stops you :)

Sorry, but I am still having a problem. The downloading scripts work run manually, but not by pressing the IDS button.

I looked at my /usr/bin/pre_download.sh and it was still only the original

I added

run-parts /mnt/free/download

as my best guess at what is supposed to be there, but it did not help.
I don't have run-parts in /usr/bin, one possible problem.

Any hints?

Benno

Mmh, looks like the patch during installation didn't work... Which Firmware version are you running? Could you post your /usr/bin/pre_download.sh?

To get things working you have to add "run-parts /etc/download" to your /usr/bin/pre_download.sh, as the main script is /etc/download/framework.

Yes, that change to pre_download.sh makes it work perfectly.

I am running version 2.12, original iLiad. I don't think iRex would be thrilled about me posting their code here. However, there is a comment line in it


# No implementation yet


after which is where new code is supposed to go -- perhaps they changed it slightly in v2?

Nice to hear that it works now, although I would like to know why the installation didn't work... I've got 2.12 running on a v2 machine too. Maybe you could PM me your pre_download.sh?

Dear Thomas,

since my pre_download.sh remained as well unchanged after installation I took the liberty to PM it as well for your information.

Meanwhile best greetings to you and all iLiad warriors.

D'oh :smack:, /usr/bin/patch is only included in Adam B.'s new busybox (http://www.mobileread.com/forums/showthread.php?t=12386) and not on a "virgin" Iliad... Shame on me, and thanks to your bug report. I will think of an alternative and update the installation package soon.

I've corrected the bug and uploaded a new installer (see first post). Simply install the new version and things should be fine :D

Thanks again for your bug reports :)

Hi Thomas,

Thank you for such great software!
Don't we need to uninstall the version 0.1 first?

Thanks,

dabon

Don't we need to uninstall the version 0.1 first?


You don't have to, it should work with the original version still installed.

If you want to uninstall the old package first, you have to do a ipkg remove autodl on your favorite command line.

Thanks a bunch!!

Dabon.

A plugin for Feedbooks.com RSS subscriptions is coming in the near future... :thumbsup:

Thomas, master, have you managed with creating the plugin? ;)

Thomas, master, have you managed with creating the plugin? ;)
Unfortunately I have exams this week and don't have the time to complete it at the moment. Additionally Feedbooks has recently changed their API (not a major change though), so I will need some time ;)

Unfortunately I have exams this week and don't have the time to complete it at the moment. Additionally Feedbooks has recently changed their API (not a major change though), so I will need some time ;)

I'll start publishing the documentation for our new API this week, and yes it's a major change: there's now dozens of endpoints.

Unfortunately I have exams this week and don't have the time to complete it at the moment. Additionally Feedbooks has recently changed their API (not a major change though), so I will need some time ;)

Oh, I see. We'll be waiting with impatience :) Good luck, men :bookworm:

I'll start publishing the documentation for our new API this week [...]
Cool, what about saturday, 11.30 AM CEST? My exams will be over then *g*

I must be dumb, by the comments this works for most people, but not for me! I unzipped the autodl-v0.2 to main memory on my iLiad 2 running 2.12 and have an entry in main memory that says: 'Install Automatic Downloader Framework' I can click this any way I like but nothing happens!

If I connect the iLiad to the PC I see a folder: 'downloader_install' which I can open and see five files including 'run.sh' etc.

Where on earth is "/usr/bin/pre_download.sh" etc. I think we are into Linux, do I need to load an SDK or something?

I would appreciate some help please!

Seadart

I guess you're missing the developer's package. Have a look at this thread (http://www.mobileread.com/forums/showthread.php?t=17342), there's a detailed howto.

Please drop me a note if this doesn't solve the problem :thumbsup:

You'll find here the plugin in order to download the "20 minutes" newspaper in french.
I haven't change the icon for now (still using 20 minutos icon).

Thanks again Thomas for your wonderful framework !

19th september 2008 - edit : I've updated the package by adding a test : if the issue is not available on the web (like during the week-end), the plugin does not delete latest downloaded issue. I did the same for the "20 minutos" plugin too.

Grégory

A plugin for Feedbooks.com RSS subscriptions is coming in the near future... :thumbsup:
Just a little update on this: Unfortunately I don't have much free time at the moment, so the Feedbooks plugin won't appear in the _very_ near future... :rolleyes: But it's still on my list!

Hello Guys,

I've finished a new plugin downloading the latest edition of the Sunday Times Software Development : http://www.sdtimes.com/content/currentissue.aspx

Each time you'll press the "synchonize button", it will try to download issue numbered in file "SundayTimes/script/next-issue-to-download.txt". On success, it increase the issue number and update that file.
At time of publication, "next-issue-to-download.txt" file contains number 200, which is june's edition.
If you wan't to download directly the latest edition, update the number in that file on this month issue.

Edit : I've just fixed a bug on a stop condition.

Enjoy,
Gregory

I'm a new Iliad user and I just discovered this thread and wanted to say thanks. :)

Since regular downloading and reading of news is one of the things I got my (second-hand) Iliad for, this fits in perfectly.

I've had a long break from the iLiad development scene - I've actually been reading books (on my iLiad) instead of playing with it :)

Then I discovered this thread, which seems like just what I think the iLiad should have had from the beginning. I am really looking forward to trying it out. I have just one question:

Why the choice of /mnt/free/download? The only way I know of accessing this (short of file manager, ssh and terminal) is by setting default USB location to internal memory and connecting through USB. I usually just plug my CF card into my reader - connecting with USB requires the "travel hub", which is at my bed (where I usually am when the iLiad runs out of power...).

Can this directory be changed easily? Or would a softlink work (I guess I can't have a softlink on my FAT CF card to /mnt/free/download, but perhaps from /mnt/free/download to /mnt/cf/download?)

Anyway, off to find usb cables and do some experimenting :)

...
Anyway, off to find usb cables and do some experimenting :)

At my first attempt I forgot the CF card and got scared when my "books" directory was emtpy :)

Second attempt: Worked perfectly! Great work all of you! Now I gotta try some more plugins.

Just a weird observation: After clicking on the iDS the first time (when my CF was ejected), and every time afterwards, the "Download history" shows "etc" and "tmp". They are clickable, too, allowing me to view the internal file system :)

Why the choice of /mnt/free/download?
This is mainly due to two reasons:
1. There is a known bug when using WiFi and the CF card at the same time, and I didn't want to run into problems
2. Everybody has /mnt/free, but not everyone has a CF card. So I chose the "common denominator", to say it in maths :-)

Can this directory be changed easily? Or would a softlink work (I guess I can't have a softlink on my FAT CF card to /mnt/free/download, but perhaps from /mnt/free/download to /mnt/cf/download?)

/mnt/free is a FAT partition too, so symlinks won't work. But you could try to change the paths in
1. /etc/download/framework and
2. /mnt/free/documents/downloader/manifest.xml

Please report if it works for you...

Just a weird observation: After clicking on the iDS the first time (when my CF was ejected), and every time afterwards, the "Download history" shows "etc" and "tmp". They are clickable, too, allowing me to view the internal file system :)

Hmm, never seen this before. The download history resides in /mnt/settings/downloadhistory and is a collection of symlinks, so you might look into that directory...

Just a weird observation: After clicking on the iDS the first time (when my CF was ejected), and every time afterwards, the "Download history" shows "etc" and "tmp". They are clickable, too, allowing me to view the internal file system :)

This is from a recent Italian localization update.

BTW Thomas, after learning about how the recent downloads are tracked, I think we should update the autodownload scripts to record recent downloads.

Finally, it's exams time again for me, and so I need some things to waste my time on ;): Here's the Feedbooks subscriptions plugin. It will download your subscribed feeds to the iLiad. Some extra installation instructions:

You will have to install Ruby (http://www.mobileread.com/forums/showthread.php?t=36117) to get it working (that's what I ported it for :D)
Unzip the attachment and change the feedbooks-subscriptions/script/config.rb file (you have to add your Feedbooks username + password there)
Transfer the directory to /mnt/free/download (as usual)

Feeds will go to your newspapers folder (read from the device settings).

Have fun!

EDIT: Feedbooks has changed something at their API (HTTP Basic auth now requires the MD5 Hash). Please download the attached feedbooks-subscriptions_v2.zip and replace your old directory!

Hello, I have an off-topic-question, I post it here because the idea might be a bit similar to your feedbook-plugin (@thomas)

I am looking for a tool, which allows me to

1. import full Webpage-Content via RSS (like the news2pdf.com-tool)
2. layout the content in a free way (if possible with own templates or stylesheets),
3. creates an PDF-output.

My Aim is to create a weekly magazin with my blog-content. There are some tools which do so (feedbooks, a more magazin-style is tabbloid.com or feedjournal.com).

I'm looking for such a tool for weeks now, and a hint would be very helpfull. Please note: I don't want an automattic tool, there are lots of them, what I want is a free Layoutfunction.

Thanks a lot!

Hi all,

I've just launched a website called www.offlinereading.com (http://www.offlinereading.com) where you can find download links (and podcast URL's) for the latest edition of many newspapers and magazines. This might simplify plugin development for Thomas' Automatic Download Tool (because you now have a 'fixed' URL for the PDF).

Anton

Here's the Feedbooks subscriptions plugin.
It will download your subscribed feeds to the iLiad.
...
Have fun!

Thanks for this Thomas, I tried it, running version 2.12, Software build number 21201.

I installed ruby 1.8.5 from your other post. And I'm getting this error - could this be the xml parser (REXML) issues mentioned by Hansel?

Your help would be appreciated.

the output from main_output.log:

Feedbooks downloader
Trying to find newspapers storage from /mnt/settings/registry/registry.xml...
Setting found: /mnt/cf/newspapers
Feeds will be stored in /mnt/cf/newspapers/Feeds
Downloading subscriptions.xml...
Parsing subscriptions.xml...


the output from main_error.log:

./feedbooks.rb:107: undefined method `each_element' for nil:NilClass (NoMethodError)

Thanks for pointing me towards this. Feedbooks changed something at their API (they now require the MD5 hash of your password...), so the script gave this error. Either download the new version from my original posting (http://www.mobileread.com/forums/showpost.php?p=324601&postcount=35) or change the following two things at your script/feedbooks.rb:

1. Add a line after line #12:
require "digest/md5"

2. Change line #33 to request.basic_auth @feedbooks_username, Digest::MD5.hexdigest(@feedbooks_password)

That's a worker, thanks alot! :D

I created a plugin for the site of the Austrian televison company ORF (www.orf.at).
Thanks to thomas and his "Tagesschau" plugin, on which this one is based.

The plugin uses the site go.orf.at, which has a puristic design that I prefer for reading on the iLiad.

Hi Thomas
I have installed the developer and mrxvt

internal memory automatic downloaders framework
when pressed dummy downloader appears
current status active
pressed button download now
nothing happens


saved guardian24 on the mmccard
dont know where to find mnt/free/download

pressed the IDC button - no downloads available
pressed news (empty)

what am i doing wrong?

please assist

Thanks in advance for your prompt reply
Ron