How to transfer only previously unretrieved RSS posts?

[lymeswold]

I posted a similar question to this on the main Calibre forum yesterday, and dwanthny kindly pointed out that it would be better addressed to the recipes forum, so here goes.

I purchased a Kindle 3 last week and I've configured Calibre to download some RSS feeds and email them to my Kindle via he free.kindle.com address.

But I would prefer each transfer to contain only the posts I've not yet seen, rather than a rolling cycle of the last n days or n posts.

Of course, that requires the datetime of the last retrieved post in each feed to be stored and used as the starting point for the next tranche.

So my question is: can Calibre do this, ie does it have access to persistent storage. If it can, are there any example recipes I could look at (I know .NET languages but not Python)? And if I have to preprocess the feeds myself before Calibre retrieves them, can I still hook into the 'retrieve - convert - email' pipeline using the Calibre command line?

Thanks for any advice.

Chris

[kovidgoyal]

Sure you can do it. Just re-implement the parse_feeds function in your recipe. AS for storage, use this

from calibre.utils.config import dynamic

history = create dict of times

dynamic['recipe_name_last_downloaded_dates'] = history

[lymeswold]

Originally Posted by kovidgoyal

Sure you can do it. Just re-implement the parse_feeds function in your recipe.

Thanks for the pointer. I've looked at the source for parse_feeds and concluded that modifying it to handle multiple feeds is beyond copying/pasting existing code and changing a few variable values, and that my Python ability is not up to modifying it any time soon. I was hoping someone might already have done something similar.

In the UK Amazon is doing a BIG push on Kindle 3. The ads are everywhere. My guess is that a lot of people will want this functionality (already I see someone has replied to my original post in the Calibre forum expressing interest).

If doing that development appeals to anyone, please say!

My alternative is to implement an intelligent feed filter/aggregator using .NET & DotNetNuke and have Calibre read from that.

[akadaedalus]

Been staring at this problem for a little while. My conclusion is that this is not really well supported.

I'm about to embark on writing a Perl script cron job that will import the scheduler.xml file, open the recipe for each scheduled feed, check the RSS URLs for any updates since the last pull, and execute ebook-convert for any newly updated feeds. Along the way it'll have to use sqlite3 to pull the custom recipes.

While all this seems it would be easier to write in a plugin using the existing APIs, I don't like Python and it would take me more time to code it up that way, and it sounds prone to breakage in new versions of calibre. Here is where I'm hoping someone more proficient with Python already accomplished all or most of the above!

Desires:
1) only generate a new book if there is an update in the RSS feed (I have a couple rarely-updated blogs I'd love to follow)
2) not have to update every single recipe I'm interested in with code to make it work (aforementioned parse_feed solution)
3) keep each feed in separate books (meaning I'm not interested in the aggregated google reader solution)
3) automate w/ cron and command line tools to avoid using the gui
4) bonus: trigger a sync to the device when plugged in, without gui (udev trickery?)

[Starson17]

Originally Posted by akadaedalus

Been staring at this problem for a little while.

If any of this has been done already, it hasn't been reported here. If it's of any help, I took Kovid's comments and had no trouble using persistent storage in several test recipes. If all you want is to not fetch previously fetched article URLs, or exit if the first article URL is unchanged it should be pretty easy. If you want to avoid the gui or avoid changing each recipe, you'll have to do something like you've suggested.

[TonytheBookworm]

Originally Posted by Starson17

If any of this has been done already, it hasn't been reported here. If it's of any help, I took Kovid's comments and had no trouble using persistent storage in several test recipes. If all you want is to not fetch previously fetched article URLs, or exit if the first article URL is unchanged it should be pretty easy. If you want to avoid the gui or avoid changing each recipe, you'll have to do something like you've suggested.

Do you mind posting a code snippet of one of your recipes that utilizes the do not fetch if first previous url hasn't changed. I would like to implement that in some of my recipes but never actually figured out how to do it other than what i mentioned in one of my post about the date check. It works sorta but the article still renders, it is just blank I was hoping to totally abort the whole thing and not even have the feed save period. maybe that isn't possible.

[marbs]

but there is a short sticky called reusable code. check that out.

[Marts]

This would be very useful indeed!

Also, for sites which are quite fast moving and only have a small number of RSS items, it would be useful to be pulling down the RSS regularly (every 30 mins, say) but only generate the ebook once - in the morning for instance.

Cheers,

M.

[Starson17]

Originally Posted by TonytheBookworm

Do you mind posting a code snippet of one of your recipes that utilizes the do not fetch if first previous url hasn't changed. I would like to implement that in some of my recipes but never actually figured out how to do it other than what i mentioned in one of my post about the date check. It works sorta but the article still renders, it is just blank I was hoping to totally abort the whole thing and not even have the feed save period. maybe that isn't possible.

Tony,

All I've done in my testing is save and retrieve data of interest (time, first article url, etc.) to/from a recipe between multiple runs. I haven't tried anything sophisticated with what I do with that data. In fact, I saw that you used exit code in a recipe and thought I might adapt that to abort the recipe when the first feed hadn't changed.

I did this sort of thing in my tests:

        now = #set current time in desired format using datetime
        first_url = # set current first_url
        previous_download_time = dynamic['recipe_name']['download_time'] 
        previous_first_url = dynamic['recipe_name']['first_url'] 
        dynamic['recipe_name']['download_time'] = now
        dynamic['recipe_name']['first_url'] = first_url

This retrieves the info about the previous recipe run and sets the new info. What you do with it is up to the recipe author. My tests just put it in print_version because that's the point where the recipe has parsed the RSS for article URLs.

[Starson17]

Originally Posted by TonytheBookworm

It works sorta but the article still renders, it is just blank I was hoping to totally abort the whole thing and not even have the feed save period. maybe that isn't possible.

Are you familiar with:
remove_empty_feeds = True
?

I vaguely recall that you get an abort if all feeds are empty.

[TonytheBookworm]

Originally Posted by Starson17

Are you familiar with:
remove_empty_feeds = True
?

I vaguely recall that you get an abort if all feeds are empty.

I have seen that flag before but had forgotten about it. Will try it in the future and see what happens thanks.