Fayetteville Observer - Fayetteville, NC USA (dups)

[amr]

How does one improve a recipe to avoid duplicate articles across a site?

Here is there recipe and it works, but an article can be in more than one sub section. I would like to download it only once.

class AdvancedUserRecipe1293623816(BasicNewsRecipe):
title = u'Fayetteville Observer'
oldest_article = 7
max_articles_per_feed = 30

feeds = [(u'Local News', u'http://fayobserver.com/CMSPages/rssNews.aspx'), (u'Life', u'http://fayobserver.com/CMSPages/rssLife.aspx'), (u'Business', u'http://fayobserver.com/CMSPages/rssBusiness.aspx'), (u'Military', u'http://fayobserver.com/CMSPages/rssMilitary.aspx'), (u'Sports', u'http://fayobserver.com/CMSPages/rssSports.aspx'),(u'Livewire',u'http://blogs.fayobserver.com/CMSPages/BlogRss.aspx?aliaspath=/livewire'),(u'Crime',u'http://blogs.fayobserver.com/CMSPages/BlogRss.aspx?aliaspath=/brooks'),(u'FaytoZ',u'http://blogs.fayobserver.com/CMSPages/BlogRss.aspx?aliaspath=/faytoz'),(u'PeoplesBusiness',u'http://blogs.fayobserver.com/CMSPages/BlogRss.aspx?aliaspath=/peoplesbusiness'),(u'910pets',u'http://blogs.fayobserver.com/CMSPages/BlogRss.aspx?aliaspath=/910pets'),(u'DadFactor',u'http://blogs.fayobserver.com/CMSPages/BlogRss.aspx?aliaspath=/dadfactor'),(u'FreeStuff',u'http://blogs.fayobserver.com/CMSPages/BlogRss.aspx?aliaspath=/free'),(u'Cheers & Jeers',u'http://fayobserver.com/cheers'),(u'TechSassy',u'http://blogs.fayobserver.com/CMSPages/BlogRss.aspx?aliaspath=/techsassy'),(u'Deviere',u'http://blogs.fayobserver.com/CMSPages/BlogRss.aspx?aliaspath=/deviere')]

[kovidgoyal]

i would advise against doing that. Sometimes it is nice to access an article from multiple feeds. And note that only the article html is downloaded multiple times. Not images/css/etc. so the bandwidth/size overhead is minimal.