RSS feeds - new posts only

[lymeswold]

I'm new to Calibre (and to book readers - just received my Kindle 3 last week). So far I've configured Calibre (what a great tool!) to download some RSS feeds and automatically email them to my free.kindle.com account.

Now I would appreciate some advice on what is and is not possible.

Google Reader on my laptop shows me which posts I've already read in each subscribed feed. I would like to duplicate that experience on the Kindle, by only sending to it when there are new posts, and then only sending the new content.

I've read through the Calibre manual and looked at a few recipes, but this doesn't look like how any of them work. Rather, they go back only a certain length of time and they limit the number of posts. The result is a lot of duplication on material arriving on the Kindle.

I looked briefly at RSS filters to preprocess feeds, but none of these appears to do what I want either. Of course, a filter would need to remember what had last been sent.

So my question is, can this be done? Is anyone doing it? Also, standard Calibre functionality appears to support a maximum automatic download frequency of once a day. Do I need to run multiple copies of Calibre (on different machines) to increase this? Or can I run Calibre from a batch file and still get the 'retrieve - convert - email' pipeline?

Thanks,

Chris

[DoctorOhh]

This should be in the Calibre Recipes sub-forum.

Originally Posted by lymeswold

Also, standard Calibre functionality appears to support a maximum automatic download frequency of once a day. Do I need to run multiple copies of Calibre (on different machines) to increase this? Or can I run Calibre from a batch file and still get the 'retrieve - convert - email' pipeline?

If you want more then one download for a given recipe in a day just make copies of the recipe and change the titles to something akin to News 8am, News 1pm, News 6pm ect... then schedule each recipe accordingly.

[pilluli]

Originally Posted by lymeswold

Google Reader on my laptop shows me which posts I've already read in each subscribed feed. I would like to duplicate that experience on the Kindle, by only sending to it when there are new posts, and then only sending the new content.

I am also quite new to calibre and kindle and I am looking for the same thing. Importing only new posts from RSS feeds and ignoring the already read. Sorry but I was also unable to find any solution.

Just wanted to let you know that you're not alone. Sorry for not being able to help!!! but let's see if anybody here can ...

Regards,

[janvanmaar]

Yes, I would also very much like frequent incremental updates.
However, even if this was supported in Calibre (which is not AFAIK) or if you implemented it, the major problem is the lack of support by Kindle. So even if Calibre downloaded incrementals only, it would still have to send the complete issues to Kindle, thus you would only save a bit of bandwidth.
Actually, even sending multiple complete issues in the same day is not handled properly by Kindle:
https://www.mobileread.com/forums/sho...09#post1217209

So to conclude, I am afraid that this is up to Amazon developers and the only way to do say hourly updates is to send the complete issues (eg using dwanthny's workaround, although adding hourly/other updates should be very simple to implement in Calibre) and then read them from the 'back issues' collection...

[oecherprinte]

I guess you would have to do some changes in your recipes to do that. You could change the parse_feeds() function to only get the articles that were added since your last access. This would require you to store the date of the last access somewhere, or you could extract the date of the file. Then you could compare the article date with the date of the last access and only retrieve the newer articles.

Here is a description of the parse_feeds function:

http://calibre-ebook.com/user_manual...ownloaded-html

[janvanmaar]

@oecherprinte: if you do that then you end up with plenty of incrementals being sent to the Kindle, all of which you would have to open and later on remove manually. I cannot see any real advantage of this over simply sending complete update issues under different names: you create the same mess that you have to clean up manually but at least you only need to open the latest file at any time to access all the latest content.

[Starson17]

Originally Posted by oecherprinte

You could change the parse_feeds() function to only get the articles that were added since your last access. This would require you to store the date of the last access somewhere

FYI, I ran some tests using code suggested by Kovid and was able to store article URLs on a first recipe run and retrieve them on a second run for comparison.

I found that I didn't really like the idea and didn't go any further. If implemented, you'd have to have all the sequence of downloaded recipes, as any given article would only appear in only one. I read intermittently (too many to read them all) and preferred having everything in the most recent book.

[oecherprinte]

@janvanmaar:

I think this is exactly what lymeswold was looking for ...

Originally Posted by lymeswold

I would like to duplicate that experience on the Kindle, by only sending to it when there are new posts, and then only sending the new content.

So he just wants to transmit the new articles.

By the way, on the kindle these incremental updates should be moved to the
dated periodicals collection and then they are automatically deleted, aren't they?

[oecherprinte]

Originally Posted by Starson17

FYI, I ran some tests using code suggested by Kovid and was able to store article URLs on a first recipe run and retrieve them on a second run for comparison.

If you would do a date comparison, you would only need a single run which would be pretty fast since you are only getting the latest articles.

[janvanmaar]

Well, Google Reader (the experience from whose usage lymeswold wanted to duplicate) displays everything at once and does not require any manual cleaning, so I anticipated the question is how to get the same behaviour on Kindle... but I agree that it is not really clear what exactly lymeswold is asking for.
I think my reply was actually bit selfish: I would not mind achieving the 'Google Reader experience' on my Kindle either... however, I find sending incrementals to it even worse than sending entire actualizations, from the reasons I mentioned above.

EDIT: I have noticed the edition to your post now:
It depends how you name the incrementals. If they get the same name then they are moved to the periodicals collection and deleted automatically - however, they are not deleted in the right order, so in my tests in was actually the latest one that was flagged for deletion as first. And the one showed in home is actually the oldest from that day.
If the names are different then they all stay in home in the right order, you can in principle read them all and then delete them all manually...
Neither case seems tempting to me. It would require proper Kindle support.

[oecherprinte]

Originally Posted by janvanmaar

Well, Google Reader (the experience from whose usage lymeswold wanted to duplicate) displays everything at once and does not require any manual cleaning, so I anticipated the question is how to get the same behaviour on Kindle... but I agree that it is not really clear what exactly lymeswold is asking for.

But you do not need manual cleaning on the kindle. If you have more than 7 issues on your device the oldest ones will be automatically deleted. The old issues are automatically moved to a "back issues" folder where they are out of the way.

See bottom of page 32 of this manual:

https://kindle.s3.amazonaws.com/Kind...%20English.pdf

I actually like the idea.

[janvanmaar]

It seems I am slow in answering - you always seem to reply or edit your reply before I reply or edit my reply
See edit of my answer above but in short: I also like the idea, unfortunately it does not work for this case.
EDIT: Part of this problem is also mentioned in this thread:
https://www.mobileread.com/forums/sho...09#post1217209

[oecherprinte]

OK. That seems to be a problem with several periodical updates in one day. Since my recipe downloads legal decisions on a 2 weekly basis, that is not really a problem for me.

When I have some time I might consider implementing this "download unread" feature for my recipe.

[janvanmaar]

Yes, less frequent updates would not suffer by these problems. I thought we were talking about frequent updates (eg latest news fetching).

[Starson17]

Originally Posted by oecherprinte

If you would do a date comparison, you would only need a single run which would be pretty fast since you are only getting the latest articles.

By "first run" and "second run" I was saying I could pass information from the first time a recipe is executed to the second time so that the second run could do something different based on what happened during the first run. Current recipes don't do this - each run is essentially independent of earlier runs. The reason I wanted to do this was to keep track of the articles retrieved previously.

I'm not sure what kind of "date comparison" you have in mind, but if you don't pass information from one recipe run to the next one, you can't do much of a comparison.