Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Recipes

Notices

Reply
 
Thread Tools Search this Thread
Old 09-30-2010, 04:39 AM   #1
lymeswold
Junior Member
lymeswold began at the beginning.
 
Posts: 3
Karma: 10
Join Date: Sep 2010
Device: Kindle 3
How to transfer only previously unretrieved RSS posts?

I posted a similar question to this on the main Calibre forum yesterday, and dwanthny kindly pointed out that it would be better addressed to the recipes forum, so here goes.

I purchased a Kindle 3 last week and I've configured Calibre to download some RSS feeds and email them to my Kindle via he free.kindle.com address.

But I would prefer each transfer to contain only the posts I've not yet seen, rather than a rolling cycle of the last n days or n posts.

Of course, that requires the datetime of the last retrieved post in each feed to be stored and used as the starting point for the next tranche.

So my question is: can Calibre do this, ie does it have access to persistent storage. If it can, are there any example recipes I could look at (I know .NET languages but not Python)? And if I have to preprocess the feeds myself before Calibre retrieves them, can I still hook into the 'retrieve - convert - email' pipeline using the Calibre command line?

Thanks for any advice.

Chris
lymeswold is offline   Reply With Quote
Old 09-30-2010, 11:41 AM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,931
Karma: 5036099
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Sure you can do it. Just re-implement the parse_feeds function in your recipe. AS for storage, use this

from calibre.utils.config import dynamic

history = create dict of times

dynamic['recipe_name_last_downloaded_dates'] = history
kovidgoyal is online now   Reply With Quote
Old 10-01-2010, 04:50 AM   #3
lymeswold
Junior Member
lymeswold began at the beginning.
 
Posts: 3
Karma: 10
Join Date: Sep 2010
Device: Kindle 3
Re: How to transfer only previously unretrieved RSS posts?

Quote:
Originally Posted by kovidgoyal View Post
Sure you can do it. Just re-implement the parse_feeds function in your recipe.
Thanks for the pointer. I've looked at the source for parse_feeds and concluded that modifying it to handle multiple feeds is beyond copying/pasting existing code and changing a few variable values, and that my Python ability is not up to modifying it any time soon. I was hoping someone might already have done something similar.

In the UK Amazon is doing a BIG push on Kindle 3. The ads are everywhere. My guess is that a lot of people will want this functionality (already I see someone has replied to my original post in the Calibre forum expressing interest).

If doing that development appeals to anyone, please say!

My alternative is to implement an intelligent feed filter/aggregator using .NET & DotNetNuke and have Calibre read from that.
lymeswold is offline   Reply With Quote
Old 10-13-2010, 07:31 PM   #4
akadaedalus
Junior Member
akadaedalus began at the beginning.
 
Posts: 9
Karma: 10
Join Date: Jun 2008
Device: Sony PRS-500
Been staring at this problem for a little while. My conclusion is that this is not really well supported.

I'm about to embark on writing a Perl script cron job that will import the scheduler.xml file, open the recipe for each scheduled feed, check the RSS URLs for any updates since the last pull, and execute ebook-convert for any newly updated feeds. Along the way it'll have to use sqlite3 to pull the custom recipes.

While all this seems it would be easier to write in a plugin using the existing APIs, I don't like Python and it would take me more time to code it up that way, and it sounds prone to breakage in new versions of calibre. Here is where I'm hoping someone more proficient with Python already accomplished all or most of the above!

Desires:
1) only generate a new book if there is an update in the RSS feed (I have a couple rarely-updated blogs I'd love to follow)
2) not have to update every single recipe I'm interested in with code to make it work (aforementioned parse_feed solution)
3) keep each feed in separate books (meaning I'm not interested in the aggregated google reader solution)
3) automate w/ cron and command line tools to avoid using the gui
4) bonus: trigger a sync to the device when plugged in, without gui (udev trickery?)
akadaedalus is offline   Reply With Quote
Old 10-13-2010, 08:39 PM   #5
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by akadaedalus View Post
Been staring at this problem for a little while.
If any of this has been done already, it hasn't been reported here. If it's of any help, I took Kovid's comments and had no trouble using persistent storage in several test recipes. If all you want is to not fetch previously fetched article URLs, or exit if the first article URL is unchanged it should be pretty easy. If you want to avoid the gui or avoid changing each recipe, you'll have to do something like you've suggested.
Starson17 is offline   Reply With Quote
Old 10-13-2010, 10:58 PM   #6
TonytheBookworm
Addict
TonytheBookworm is on a distinguished road
 
TonytheBookworm's Avatar
 
Posts: 264
Karma: 62
Join Date: May 2010
Device: kindle 2, kindle 3, Kindle fire
Quote:
Originally Posted by Starson17 View Post
If any of this has been done already, it hasn't been reported here. If it's of any help, I took Kovid's comments and had no trouble using persistent storage in several test recipes. If all you want is to not fetch previously fetched article URLs, or exit if the first article URL is unchanged it should be pretty easy. If you want to avoid the gui or avoid changing each recipe, you'll have to do something like you've suggested.
Do you mind posting a code snippet of one of your recipes that utilizes the do not fetch if first previous url hasn't changed. I would like to implement that in some of my recipes but never actually figured out how to do it other than what i mentioned in one of my post about the date check. It works sorta but the article still renders, it is just blank I was hoping to totally abort the whole thing and not even have the feed save period. maybe that isn't possible.
TonytheBookworm is offline   Reply With Quote
Old 10-14-2010, 03:16 AM   #7
marbs
Zealot
marbs began at the beginning.
 
Posts: 122
Karma: 10
Join Date: Jul 2010
Device: nook
i am not sure this can be adapted easly

but there is a short sticky called reusable code. check that out.
marbs is offline   Reply With Quote
Old 10-14-2010, 09:23 AM   #8
Marts
Junior Member
Marts began at the beginning.
 
Posts: 3
Karma: 10
Join Date: Oct 2010
Device: Kindle 3
This would be very useful indeed!

Also, for sites which are quite fast moving and only have a small number of RSS items, it would be useful to be pulling down the RSS regularly (every 30 mins, say) but only generate the ebook once - in the morning for instance.

Cheers,

M.
Marts is offline   Reply With Quote
Old 10-14-2010, 10:44 AM   #9
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by TonytheBookworm View Post
Do you mind posting a code snippet of one of your recipes that utilizes the do not fetch if first previous url hasn't changed. I would like to implement that in some of my recipes but never actually figured out how to do it other than what i mentioned in one of my post about the date check. It works sorta but the article still renders, it is just blank I was hoping to totally abort the whole thing and not even have the feed save period. maybe that isn't possible.
Tony,

All I've done in my testing is save and retrieve data of interest (time, first article url, etc.) to/from a recipe between multiple runs. I haven't tried anything sophisticated with what I do with that data. In fact, I saw that you used exit code in a recipe and thought I might adapt that to abort the recipe when the first feed hadn't changed.

I did this sort of thing in my tests:
Code:
        now = #set current time in desired format using datetime
        first_url = # set current first_url
        previous_download_time = dynamic['recipe_name']['download_time'] 
        previous_first_url = dynamic['recipe_name']['first_url'] 
        dynamic['recipe_name']['download_time'] = now
        dynamic['recipe_name']['first_url'] = first_url
This retrieves the info about the previous recipe run and sets the new info. What you do with it is up to the recipe author. My tests just put it in print_version because that's the point where the recipe has parsed the RSS for article URLs.
Starson17 is offline   Reply With Quote
Old 10-14-2010, 10:58 AM   #10
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by TonytheBookworm View Post
It works sorta but the article still renders, it is just blank I was hoping to totally abort the whole thing and not even have the feed save period. maybe that isn't possible.
Are you familiar with:
remove_empty_feeds = True
?

I vaguely recall that you get an abort if all feeds are empty.
Starson17 is offline   Reply With Quote
Old 10-14-2010, 07:37 PM   #11
TonytheBookworm
Addict
TonytheBookworm is on a distinguished road
 
TonytheBookworm's Avatar
 
Posts: 264
Karma: 62
Join Date: May 2010
Device: kindle 2, kindle 3, Kindle fire
Quote:
Originally Posted by Starson17 View Post
Are you familiar with:
remove_empty_feeds = True
?

I vaguely recall that you get an abort if all feeds are empty.
I have seen that flag before but had forgotten about it. Will try it in the future and see what happens thanks.
TonytheBookworm is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
RSS feeds - new posts only lymeswold Calibre 25 11-17-2010 06:06 PM
iPad Why I decided to reserve an iPad when I previously had no interest.... DougFNJ Apple Devices 36 04-02-2010 11:32 PM
Libprs500- error with adding previously converted books bluesnews Calibre 4 04-08-2008 11:06 PM
Question on previously purchased content sana Workshop 7 03-09-2008 07:19 AM


All times are GMT -4. The time now is 12:39 PM.


MobileRead.com is a privately owned, operated and funded community.