Time Calculation off by 14 hours

[ChappyOnIce]

Is anyone else running across this? I've been trying to understand why I have to download 2 days worth of news to get every article in a 24 hour period and it seems that the time used by Calibre to determine if an article is too old is off by 14 hours. Here's a snippet of a log from a job this morning:

Skipping article Vanek expects wild Minnesota homecoming (Sat, 05 Mar, 2011 08:40) from feed Sabres & NHL as it is too old.
Skipping article Will Enroth start? Mum's the word (Fri, 04 Mar, 2011 09:55) from feed Sabres & NHL as it is too old.
Skipping article Vanek, Stafford look to snap slumps (Fri, 04 Mar, 2011 09:55) from feed Sabres & NHL as it is too old.

Here's the RSS feed (Eastern time):



Vanek expects wild Minnesota homecoming
Yesterday, March 05, 2011, 10:40:57 PM
Sabres forward expects to have 40-50 fans in attendance at XCel Energy Center

Will Enroth start? Mum's the word
Friday, March 04, 2011, 11:55:39 PM
Goalie recalled from Portland and could see action today against Flyers

Vanek, Stafford look to snap slumps
Friday, March 04, 2011, 11:55:18 PM
Buffalo's top scorers having a hard time finding the back of the net.

Any suggestions? This was run this morning at 9am Eastern, with a 1 day old flag. I have confirmed from other articles that it does use a 24 hour period as there are articles that cross the date/day barrier.

Thanks!

[ChappyOnIce]

Oddly enough, if I just do a basic recipe with the following feed:

http://www.buffalonews.com/sports/sa...ontentId=77944

and set the oldest article to "1 day" in the dialog box, it knows the article date. Compare the log entries below to the ones from my original post.

Skipping article Will Enroth start? Mum's the word (Fri, 04 Mar, 2011 23:55) from feed Sabres as it is too old.
Skipping article Vanek, Stafford look to snap slumps (Fri, 04 Mar, 2011 23:55) from feed Sabres as it is too old.

Could I be parsing out some data needed for the calculation? Is it determining article inclusion from the RSS feed page or from the actual article HTML? Is there a different method called for parsing feeds when using the advanced recipe instead of the basic?

UPDATED: I just took that basic recipe, switched to advanced mode, didn't add a single line, and ran the recipe. The date was off again. Must be a different parsing of the RSS feed when you switch to advanced mode which is causing the issue.

[ChappyOnIce]

After looking into this more myself, I'm thinking that it could be a problem with a bogus time format on the Buffalo News Atom feed. Here's what the feed looks like:

            <item> 
              <title>Mold risk could stall courthouse opening</title> 
              <link>http://www.buffalonews.com/city/communities/downtown/article361082.ece</link> 
              <description>Mysterious moisture problem in building's walls poses health hazard, concerns federal officials</description> 
              <pubDate>Tue, 08 Mar 2011 10:08:23 -0500</pubDate> 
              <guid>http://www.buffalonews.com/city/communities/downtown/article361082.ece</guid>	
            </item>

I noticed on the feedparser site that the date listed in the pubDate tag is not a standard date. Therefore I believe this issue may have something to do with the UTC conversion of the date? The feedparser site does not show this format as being a recognized format and it's probably spitting out something that is off 14 hours as a result. Thoughts and guidance? It seems like a custom date handler would have to be written.

I'm hoping someone can chime in on if they think I'm on the right or wrong path. I know there has been some discussion about dates and conversion (Kindle formats, etc.), but this clearly has to do with a recipe process. Has anyone implemented a custom date handler in another recipe I can look at?

Thanks,

Chappy

[kovidgoyal]

Yes, this would be because of the non standard time format. You would have to override the parse_feeds function to implement your own parsing logic.

[ChappyOnIce]

Thanks Kovid. Can you tell me which module/function is responsible for the job output text of "Skipping article....from feed....as it is too old."? Is that the "feed_from_xml" function that's being called from the built-in parse_feeds?

[kovidgoyal]

yes, look in feeds.__init__