Bug in web/feeds/__init__ processing feed article date

[nickredding]

There is a problem in parsing RSS feeds where the article dates are not being processed properly. In the routine parse_article the article date is being extracted as item.get('date_parsed', time.gmtime()) which is returning the current time because the feed/article key 'date_parsed' is not found.

In two major RSS feeds (NYTimes and Globe and Mail) the feed/article date key is 'published_parsed'. I realize there are variants of RSS feed formats but I believe (based on looking at feedparser.py) that 'published_parsed' is a valid, non-deprecated key.

As a result, any RSS feed-based recipes that use feeds with the 'published_parsed' key are not obeying oldest_article restrictions.

I'm not sure what the best fix is--probably to extract using 'published_parsed' if the 'date_parsed' key isn't there (or vice versa) to avoid breaking feeds that use 'date_parsed'.

[jumafl]

Could this be the same issue that is causing the oldest article function not to skip old articles in basic recipes? The issue started with Calibre 0.9.23.

For example, I have a basic recipe to pull RSS feeds for betanews.com. The fields in this basic recipe are as follows (to recreate this issue you will need to create a basic recipe and enter these fields).

class BasicUserRecipe1363558652(AutomaticNewsRecipe):
title = u'Beta News'
oldest_article = 1
max_articles_per_feed = 100
auto_cleanup = True
feeds = [(u'Top Stories', u'http://feeds.betanews.com/bn')]

Using Calibre 0.9.21 this basic recipe correctly returns only articles posted in the last 1 day. Using Calibre 0.9.23, this recipe returns all articles on "http://feeds.betanews.com/bn" page. The oldest is currently 4 days old.

The same issue is happening with my other basic recipes. All were running correctly until I installed 0.9.23.

[nickredding]

^I don't know when this problem arose but yes, quite likely this is why.

[jumafl]

I can't say it is related but the Calibre 0.9.23 change log shows this bug fix:

News download: Update the library used to parse RSS feeds.
Closes tickets: 1152852

[kovidgoyal]

http://bazaar.launchpad.net/~kovid/c...revision/14619