Avoid confirmation pages before the news | problem with Adnkronos rss feeds

[heYooh]

Hello everyone, I want to report a problem with the download of the rss feeds from the Italian news website Adnkronos.com. It used to work very good until the last few months. Now Calibre makes an ebook which only cointains the index and when you open the chapters you don't find the articles but only their urls.
I checked the recipe file and all the web addresses are correct because they match the ones on this page http://www.adnkronos.com/rss. The only thing that I found out is that when you open an article (for example, pick one from this page http://rss.feedsportal.com/c/32375/f/448341/index.rss) then you are not linked to the actual article, but instead you are redirected to a page which asks your confirmation click before enabling you to see the article. So I think this must be the cause of the problem, because it may prevent Calibre from downloading the articles. It's probably a measure implemented only recently by Adnkronos.

Anyway, are you aware of any method to bypass these confirmation pages?

[kovidgoyal]

You need to implement get_obfuscated_article() in your recipe. You can use it to bypass ad pages.

[heYooh]

Hi Kovid, it's such a pleasure to talk to you! It's my first time here and first of all I want to thank you for all the great work you've done on Calibre, as it is a wonderful software and, as I can see, it keeps improving continuously.

About this "get_obfuscated_article()" function, is it documented somewhere? I would be glad to fix this recipe and send it back to you so that you can distribute it in the next updates

[kovidgoyal]

The entire news fetch API including that function is documented in the User Manual, at http://manual.calibre-ebook.com/news_recipe.html

[heYooh]

I've set articles_are_obfuscated = True but I don't understand how to use the def get_obfuscated_article(self, url): part.

The existing code is:

def get_article_url(self, article):
        link = article.get('id', article.get('guid', None))
        return link

[kovidgoyal]

Look in some of the other builtin recipes for examples of its use, for example, forbes india

The diea is simply that you download the article, if it is an ad, you follow the link to the real article, then save the real article html in a temporary file and return the path thto thetemp file.

If it is not an add, simply return the original link.

[heYooh]

Thanks for the hint, I've given a look at the recipe of Forbes India. I'm sorry but I still can't understand how to adapt that code to my case. I'm not a programmer and my understanding of code is very limited

[heYooh]

If you or any other competent person around could give a look at this recipe and try to fix it, I would appreciate it a lot.