Irish Times - Recipe Problem

[leo738]

Hello All,

I'm currently running calibre on a Beagleboard XM (http://en.wikipedia.org/wiki/BeagleBoard). Because it's based on ARM architecture I'm stuck using version 0.7.4 of Calibre unless I want to do a full cross-compilation.

Anyway I'm having problems with the Irish Times recipe. I grabbed the recipe from v 0.8.1 of Calibre, here's an excerpt of the important bits:

('Ireland', 'http://rss.feedsportal.com/c/851/f/10845/index.rss'),
('World', 'http://rss.feedsportal.com/c/851/f/10846/index.rss'),
('Finance', 'http://rss.feedsportal.com/c/851/f/10847/index.rss'),
('Features', 'http://rss.feedsportal.com/c/851/f/10848/index.rss'),
('Sport', 'http://rss.feedsportal.com/c/851/f/10849/index.rss'),
('Opinion', 'http://rss.feedsportal.com/c/851/f/10850/index.rss'),
('Letters', 'http://rss.feedsportal.com/c/851/f/10851/index.rss'),
('Magazine', 'http://www.irishtimes.com/feeds/rss/newspaper/magazine.rss'),
('Health', 'http://rss.feedsportal.com/c/851/f/10852/index.rss'),
('Education & Parenting', 'http://rss.feedsportal.com/c/851/f/10853/index.rss'),
('Motors', 'http://rss.feedsportal.com/c/851/f/10854/index.rss'),
('An Teanga Bheo', 'http://www.irishtimes.com/feeds/rss/newspaper/anteangabheo.rss'),
('Commercial Property', 'http://www.irishtimes.com/feeds/rss/newspaper/commercialproperty.rss'),
('Science Today', 'http://www.irishtimes.com/feeds/rss/newspaper/sciencetoday.rss'),
('Property', 'http://www.irishtimes.com/feeds/rss/newspaper/property.rss'),
('The Tickets', 'http://www.irishtimes.com/feeds/rss/newspaper/theticket.rss'),
('Weekend', 'http://www.irishtimes.com/feeds/rss/newspaper/weekend.rss'),
('News features', 'http://www.irishtimes.com/feeds/rss/newspaper/newsfeatures.rss'),
('Obituaries', 'http://www.irishtimes.com/feeds/rss/newspaper/obituaries.rss'),

However it's still not fully correct. The 'Frontpage' section is ok but most of the other sections appear as:

click here to continue to article
cliquez ici pour lire l'article
weiter zum Artikel
clicca qui per visualizzare l'articolo weiter zum Artikel
ir a la noticia
klik hier om door te gaan naar het artikel
Yazıya devam etmek için tıklayın
Перейти к статье
继续阅读文章，请点击这里

Strangely enough I've experienced no problems with 'The Economist' or 'The Week'.

Does anybody have any ideas what might be causing this?

Many thanks,

Leo

[Starson17]

Originally Posted by leo738

I'm currently running calibre on a Beagleboard XM (http://en.wikipedia.org/wiki/BeagleBoard). Because it's based on ARM architecture I'm stuck using version 0.7.4 of Calibre unless I want to do a full cross-compilation.

Cool!

Anyway I'm having problems with the Irish Times recipe.
Strangely enough I've experienced no problems with 'The Economist' or 'The Week'.

Does anybody have any ideas what might be causing this?

I get the same thing with my FireFox browser that you are seeing with Calibre's recipe system. I block flash and scripts. When I go to that site, it hangs - telling me it's trying to display an advertisement and the script is not running. You'd need to rewrite the recipe to fix this.

[phiznlil]

I have seen this occasionally, usually it is only one story though at the beginning of a section. Mine downloaded at 7am this morning and I see what you are seeing only for the first story under "Ireland", I am not sure what the cause is though.

[oneillpt]

Originally Posted by leo738

Anyway I'm having problems with the Irish Times recipe. I grabbed the recipe from v 0.8.1 of Calibre, here's an excerpt of the important bits:

Here is a fix for now, pending any possible fix from the authors. I've left the line replaced as a comment. No guarantee that this fix will continue to work - this fix is based on the structure of the redirected urls as of today, and this may change again in future. If a change is needed in the next few days I'll post it on this thread, otherwise if a change is needed later I'll post it as a new thread. (Your "important bits" are not in fact the real "important bits" here. The print_version routine, just after the feeds, is where modification was needed).

Spoiler:

__license__  = 'GPL v3'
__copyright__ = "2008, Derry FitzGerald. 2009 Modified by Ray Kinsella and David O'Callaghan, 2011 Modified by Phil Burns"
'''
irishtimes.com
'''
import re

from calibre.web.feeds.news import BasicNewsRecipe

class IrishTimes(BasicNewsRecipe):
    title          = u'The Irish Times'
    encoding  = 'ISO-8859-15'
    __author__    = "Derry FitzGerald, Ray Kinsella, David O'Callaghan and Phil Burns"
    language = 'en_IE'
    timefmt = ' (%A, %B %d, %Y)'


    oldest_article = 1.0
    max_articles_per_feed  = 100
    no_stylesheets = True
    simultaneous_downloads= 5

    r = re.compile('.*(?P<url>http:\/\/(www.irishtimes.com)|(rss.feedsportal.com\/c)\/.*\.html?).*')
    remove_tags    = [dict(name='div', attrs={'class':'footer'})]
    extra_css      = 'p, div { margin: 0pt; border: 0pt; text-indent: 0.5em } .headline {font-size: large;} \n .fact { padding-top: 10pt  }'

    feeds          = [
                      ('Frontpage', 'http://www.irishtimes.com/feeds/rss/newspaper/index.rss'),
                      ('Ireland', 'http://www.irishtimes.com/feeds/rss/newspaper/ireland.rss'),
                      ('World', 'http://www.irishtimes.com/feeds/rss/newspaper/world.rss'),
                      ('Finance', 'http://www.irishtimes.com/feeds/rss/newspaper/finance.rss'),
                      ('Features', 'http://www.irishtimes.com/feeds/rss/newspaper/features.rss'),
                      ('Sport', 'http://www.irishtimes.com/feeds/rss/newspaper/sport.rss'),
                      ('Opinion', 'http://www.irishtimes.com/feeds/rss/newspaper/opinion.rss'),
                      ('Letters', 'http://www.irishtimes.com/feeds/rss/newspaper/letters.rss'),
                      ('Magazine', 'http://www.irishtimes.com/feeds/rss/newspaper/magazine.rss'),
                      ('Health', 'http://www.irishtimes.com/feeds/rss/newspaper/health.rss'),
                      ('Education & Parenting', 'http://www.irishtimes.com/feeds/rss/newspaper/education.rss'),
                      ('Motors', 'http://www.irishtimes.com/feeds/rss/newspaper/motors.rss'),
                      ('An Teanga Bheo', 'http://www.irishtimes.com/feeds/rss/newspaper/anteangabheo.rss'),
                      ('Commercial Property', 'http://www.irishtimes.com/feeds/rss/newspaper/commercialproperty.rss'),
                      ('Science Today', 'http://www.irishtimes.com/feeds/rss/newspaper/sciencetoday.rss'),
                      ('Property', 'http://www.irishtimes.com/feeds/rss/newspaper/property.rss'),
                      ('The Tickets', 'http://www.irishtimes.com/feeds/rss/newspaper/theticket.rss'),
                      ('Weekend', 'http://www.irishtimes.com/feeds/rss/newspaper/weekend.rss'),
                      ('News features', 'http://www.irishtimes.com/feeds/rss/newspaper/newsfeatures.rss'),
                      ('Obituaries', 'http://www.irishtimes.com/feeds/rss/newspaper/obituaries.rss'),
                    ]


    def print_version(self, url):
        if url.count('rss.feedsportal.com'):
            #u = url.replace('0Bhtml/story01.htm','_pf0Bhtml/story01.htm')
            u = url.find('irishtimes')
            u = 'http://www.irishtimes.com' + url[u + 12:]
            u = u.replace('0C', '/')
            u = u.replace('A', '')
            u = u.replace('0Bhtml/story01.htm', '_pf.html')
        else:
            u = url.replace('.html','_pf.html')
        return u

    def get_article_url(self, article):
        return article.link

[Starson17]

Originally Posted by phiznlil

I have seen this occasionally, usually it is only one story though at the beginning of a section. Mine downloaded at 7am this morning and I see what you are seeing only for the first story under "Ireland", I am not sure what the cause is though.

It's trying to use scripting to show you an ad. The recipe would have to be written to bypass the whole process.

[oneillpt]

Originally Posted by Starson17

It's trying to use scripting to show you an ad. The recipe would have to be written to bypass the whole process.

We posted almost simultaneously. The fix I posted builds the onward link to the article and uses it as the "printable version" of the page with the ad to which you are directed from the rss feed.

[phiznlil]

This seems to be working nicely for now

[markvdvelde]

My feeling is it mostly happens when you're downloading from multiple sources simultaneously. If you schedule downloading news with a few minutes in between each source, the result might be better. That's my experience at least.

[Starson17]

Originally Posted by oneillpt

The fix I posted builds the onward link to the article and uses it as the "printable version" of the page with the ad to which you are directed from the rss feed.

Great! I'm glad someone tackled it and provided a fix.

[leo738]

Many thanks for the fix, it's been working for the last 2 days at least!

Leo

[leo738]

The latest fix seems to be working consistently now but I've noticed that the Irish Times occasionally posts the same story under multiple RSS links. Is there some way prevent the same story been included multiple times?

Thanks,

Leo