RSS feeds - new posts only

[janvanmaar]

@Starson17: You could just look at the date of the last run and exclude all articles with date older than that - I assume at least that this was meant by 'date comparison'.

[Starson17]

Originally Posted by janvanmaar

@Starson17: You could just look at the date of the last run and exclude all articles with date older than that - I assume at least that this was meant by 'date comparison'.

Yes, but to do that you need to know the date of the last run, and that requires storing that date somewhere during the previous run and fetching it during the current run to do the comparison. That's exactly what my comment was about - passing data from one recipe run to the next one.

[Starson17]

I went back and looked at my original post. I suspect he was originally commenting on the statement that in my tests I stored article URLs for comparison. The reason I did it that way and not only by date comparison (I actually did it both ways) was to also solve the problem of duplicate articles seen in many recipes where the same article is often listed in several feeds. The feed on Energy and the feed on Politics might have the same article listed about the new Energy Bill. I was thinking about how to solve both problems simultaneously.

[oecherprinte]

Originally Posted by Starson17

Yes, but to do that you need to know the date of the last run, and that requires storing that date somewhere during the previous run and fetching it during the current run to do the comparison. That's exactly what my comment was about - passing data from one recipe run to the next one.

My idea was to look for the latest file that was generated for the feed from calibre. Then you could look up the date at which this file was generated which you can retrieve from the filesystem. I have not really looked at it, but this way you could avoid storing data in a separate history file for the feed. I am sure there is some python command to retrieve the date of a file.


Of course, the duplicate article problem cannot be solved that way.

[Starson17]

Originally Posted by oecherprinte

My idea was to look for the latest file that was generated for the feed from calibre. Then you could look up the date at which this file was generated which you can retrieve from the filesystem. I have not really looked at it, but this way you could avoid storing data in a separate history file for the feed. I am sure there is some python command to retrieve the date of a file.

That will often work, but not always. If the file is being deleted when sent to the reader, there's no local file to get a date from. Another issue is that sometimes the articles in a feed have no date, so you can't compare article dates to the file date. Still another issue with dates is that the article date isn't necessarily tied to the date you parsed the feed. Articles with older dates are sometimes added to feeds.

Regardless of all that, I eventually decided I didn't like stripping articles, particularly since it had to be done at the individual recipe level. You'd either have two of every recipe, or some recipes would have this and some wouldn't. A feature like this should probably be implemented at a higher level. I was interested in how it could be done, but when I looked at it closely, it wasn't a feature I actually wanted.

[oecherprinte]

Originally Posted by Starson17

That will often work, but not always. If the file is being deleted when sent to the reader, there's no local file to get a date from. Another issue is that sometimes the articles in a feed have no date, so you can't compare article dates to the file date. Still another issue with dates is that the article date isn't necessarily tied to the date you parsed the feed. Articles with older dates are sometimes added to feeds.

Yup. It will also be a problem with different time zones even if the article dates are correct.

Originally Posted by Starson17

Regardless of all that, I eventually decided I didn't like stripping articles, particularly since it had to be done at the individual recipe level. You'd either have two of every recipe, or some recipes would have this and some wouldn't. A feature like this should probably be implemented at a higher level. I was interested in how it could be done, but when I looked at it closely, it wasn't a feature I actually wanted.

It would be nice if this would be supported by calibre itself and just click a button in the GUI. I kind of solved this problem by creating a popup window in my recipe which prompts the user to input how many feeds should be included. In my specific case (for legal decisions), the articles are grouped in dated feeds and so I have to select the number of feeds.

[obiwan]

I have calibre running on my server to get news every day and in works great.

As others have pointed out, it would be great to receive only feeds with new content. Could this be simply solved by hashing downloaded rss, comparing hash to a stored hash and only download and send rss to kindle if a change is detected?

This sounds easy and would have great impact.

[Starson17]

Originally Posted by obiwan

Could this be simply solved by hashing downloaded rss, comparing hash to a stored hash and only download and send rss to kindle if a change is detected?

No, you can't do it this way. Recipes vary, but most have a date or something else that changes and would change the hash. There are other ways to do it, such as date comparison and article URL history. The problem is mostly finding a developer who wants this enough to do the work.

[obiwan]

Ah well... I'm Java / c# developer. Out of curiosity I downloaded sources and it seems python is... well... different 8~(

Anyway, I see fetching is done in web/fetcher. Could you point me in the direction where this feature should be implemented?

I'm not promising anything, but I am a dev who really wants this feature :P

[Starson17]

Originally Posted by obiwan

Ah well... I'm Java / c# developer. Out of curiosity I downloaded sources and it seems python is... well... different 8~(

Anyway, I see fetching is done in web/fetcher. Could you point me in the direction where this feature should be implemented?

I'm not promising anything, but I am a dev who really wants this feature :P

calibre/web/feeds/news.py is where I'd probably start.
Perhaps calibre/web/feeds/feedparser.py as well. One of your first questions will be where to store info for whatever kind of comparison you want to do.

This is how I stored the last time a recipe of "recipe_name" ran and the URL of an article retrieved on that run:

        url = last_downloaded_article_url
        now = datetime.datetime.now()
        dynamic['recipe_name']['last_time'] = now
        dynamic['recipe_name']['last_url'] = url

You retrieve it with:

        last_time_this_recipe_ran = dynamic['recipe_name']['last_time']

Have fun.
Edit:
You may need to import pickle and open/load dynamic.pickle, which is where this sort of recipe related history seems to be kept.

[obiwan]

Ok, this doesn't seem to be so hard, at least to some degree

I added unique_id to Article class:

def unique_id(self):
md5=hashlib.md5()
md5.update(self.id)
md5.update(self.title)
md5.update(self.url)
return md5.hexdigest()

Then, in news.py after line 920 I added:
if a==0:
last_article=dynamic['recipe_'+self.title+'last_article']
if last_article is not None:
#print last_article
if last_article==article.unique_id():
print " Nothing to do"
raise ValueError('No articles found, aborting')
dynamic['recipe_'+self.title+'last_article']=article.unique_id()

The good: yeey, if the last article is the last article we downloaded processing is aborted

The bad: I see no other (easy) option than throwing an exception, which of course propagates to UI. Any idea where / how to silently handle it?