Fetching The Australian

[Javed]

It seems The Australian (under English AU) is unable to be downloaded at the moment from the recipes in Calibre. As it tries to download each article it says they are failing. At the end I finish up with a 0.0MB newspaper with just the initial hyperlinks page.

Are others that try to get The Australian having the same problem at the moment?

[AprilHare]

Originally Posted by Javed

It seems The Australian (under English AU) is unable to be downloaded at the moment from the recipes in Calibre. As it tries to download each article it says they are failing. At the end I finish up with a 0.0MB newspaper with just the initial hyperlinks page.

Are others that try to get The Australian having the same problem at the moment?

Will test later.
A little offtopic: I wonder if The Oz will make it harder to access their content in the future? It is a News Corporation newspaper after all, and Rupert Murdoch has been threatening to make people pay for all content..

[kovidgoyal]

Originally Posted by AprilHare

Will test later.
A little offtopic: I wonder if The Oz will make it harder to access their content in the future? It is a News Corporation newspaper after all, and Rupert Murdoch has been threatening to make people pay for all content..

The WSJ for example is password protected, but calibre can handle that easily.

[evanmaastrigt]

I had a quick look, and it seems their website changed. But more important, the articlelinks from their feed are redirected twice; the last one points to a local path. It looks like Calibre chokes on that, which is weird because when I add the following to the recipe

    def get_article_url(self, article) :
        br = self.get_browser()
        br.open(article.link).read()
        #print br.geturl()

        return br.geturl()

articles are downloaded all right. But this makes it freaking slow. It there another workaround, anyone?

Also the articles are now empty, but that is because the CSS classes changed

[kovidgoyal]

Hmm, the download code should deal with redirects automatically, you shouldn't have to load the article first and then return the url.

[Javed]

Thanks for the replies so far. So I guess it isn't just me then.

[evanmaastrigt]

Originally Posted by kovidgoyal

Hmm, the download code should deal with redirects automatically, you shouldn't have to load the article first and then return the url.

That's what I thought too. But the download fails without this trick. Try it for yourself.
The following also works:

    
    def get_article_url(self, article):
        return article.id

which is just as weird.

Anyway, while waiting for the men in white coats to arrive, I cooked up the following

theoz.zip

With all feeds enabled it downloads in just under 7 minutes, but YMMV

[Javed]

Evanmaastrigt

Thanks for the recipe! Works really well. Download was about 8 minutes for me.