MobileRead Forums - View Single Post - Rules for mediapart.fr and rue89.com (french news websites)

Metapioca · 08-25-2013, 08:47 AM

Just improved the rule on github: it could miss some articles.

Indeed, I already had the problem before when getting the date from the article links, so the fix was ready: it seems the date in the rss feed is the creation date (when the journalist started writing the article) instead of the publication date. So any article that took more than 2 days to write would be marked as too old and not included even if it was just published.

So now I get all links, and download the article header to get the date before deciding if it is too old or not. A tiny bit slower but we won't miss important articles !

Code still at:
https://github.com/AltGr/Calibre-fre...diapart.recipe

08-25-2013, 08:47 AM	#18
Metapioca Member Posts: 18 Karma: 10 Join Date: Sep 2010 Device: Kindle 3 3G intl	Just improved the rule on github: it could miss some articles. Indeed, I already had the problem before when getting the date from the article links, so the fix was ready: it seems the date in the rss feed is the creation date (when the journalist started writing the article) instead of the publication date. So any article that took more than 2 days to write would be marked as too old and not included even if it was just published. So now I get all links, and download the article header to get the date before deciding if it is too old or not. A tiny bit slower but we won't miss important articles ! Code still at: https://github.com/AltGr/Calibre-fre...diapart.recipe