How to handle RSS sites with redirections (cgi-bin)

[Bortolotto]

Hi Buddies!

I was wondering if somebody knows how to handle RSS links with redirection (cgi-bin), like brazilian's website "IDG Now!". Here is one example:

1 - The index page: http://rss.idgnow.com.br/c/32184/f/499640/index.rss

2 - For instance, a link to one article with redirection: http://rss.idgnow.com.br/c/32184/f/4...SS/story01.htm

3 - The redirected news page: http://idgnow.uol.com.br/internet/20...o-de-usuarios/

4 - The printable version: http://idgnow.uol.com.br/internet/20...iciaPrint_view

Thank you!

[kovidgoyal]

That looks like it is using a url mangling scheme commonly used by some rss feed aggregators. Look at the builtin toi.recipe or ilsole24ore.recipe for examples of handling it.

[Bortolotto]

Thank you very much Kovid!

I followed your advice and read ilsole24ore.recipe. Based that I made the recipe below (it's attached too), and it's working pretty well.

The source is the IDG Now! Brazil, an affiliate of International Data Group Inc.
I hope my recipe could be usefull!

Spoiler:

from calibre.web.feeds.news import BasicNewsRecipe

class IDGNow(BasicNewsRecipe):
title = 'IDG Now!'
__author__ = 'Diniz Bortolotto'
description = 'Posts do IDG Now!'
oldest_article = 7
max_articles_per_feed = 20
encoding = 'utf8'
publisher = 'Now!Digital Business Ltda.'
category = 'technology, telecom, IT, Brazil'
language = 'pt_BR'
publication_type = 'technology portal'
use_embedded_content = False
extra_css = '.headline {font-size: x-large;} \n .fact { padding-top: 10pt }'

def get_article_url(self, article):
link = article.get('link', None)
if link is None:
return article
if link.split('/')[-1]=="story01.htm":
link=link.split('/')[-2]
a=['0B','0C','0D','0E','0F','0G','0I','0N' ,'0L0S','0A','0J3A']
b=['.' ,'/' ,'?' ,'-' ,'=' ,'&' ,'_','.com','www.','0',':']
for i in range(0,len(a)):
link=link.replace(a[i],b[i])
link=link.split('&')[-3]
link=link.split('=')[1]
link=link + "/IDGNoticiaPrint_view"
return link

feeds = [
(u'Ultimas noticias', u'http://rss.idgnow.com.br/c/32184/f/499640/index.rss'),
(u'Computa\xe7\xe3o Corporativa', u'http://rss.idgnow.com.br/c/32184/f/499643/index.rss'),
(u'Carreira', u'http://rss.idgnow.com.br/c/32184/f/499644/index.rss'),
(u'Computa\xe7\xe3o Pessoal', u'http://rss.idgnow.com.br/c/32184/f/499645/index.rss'),
(u'Internet', u'http://rss.idgnow.com.br/c/32184/f/499646/index.rss'),
(u'Mercado', u'http://rss.idgnow.com.br/c/32184/f/419982/index.rss'),
(u'Seguran\xe7a', u'http://rss.idgnow.com.br/c/32184/f/499647/index.rss'),
(u'Telecom e Redes', u'http://rss.idgnow.com.br/c/32184/f/499648/index.rss')
]

reverse_article_order = True