The Economist recipe does not work anymore

[NSILMike]

It has worked and still works for me. I am in Massachusetts, USA.
Windows 7 Ultimate, 64-bit.

[Parisien]

I'm also in France, and it's not working.

[parizer]

I have the same problem, the Economist recipe doesn't work from a French IP. However, I could download it without a problem with a Dutch IP (via VPN). I don't see any differences in the website layout, but I didn't do a thorough comparison.

[kovidgoyal]

Somebody that knows how to code and with a computer in France is going to have to debug this.

[bbrissy]

Same here, I am in France and it s not working anymore
Big thanks to the guy who will fix this !

[jimmyjoey]

I am also in France and most unhappy to report the Economist is not working for me anymore either. No coding ability, unfortunately.

[aritafari]

Originally Posted by jimmyjoey

I am also in France and most unhappy to report the Economist is not working for me anymore either. No coding ability, unfortunately.

I am in Los Angeles, CA and it continues to work without issues.

[fstulp]

Same problem for me, from France. I don't think it is a redesign of the Economist. Rather, in France, one must "agree and proceed" to use cookies before actually entering the site. This confirmation is probably not necessary in the US, where laws are more lenient about requiring sites to ask users about cookies. So in the US it still works. That's my working hypothesis (could someone confirm that by going on the site in the US? remove the cookie first though). What surprises me though, is that it still works in the Netherlands (see post by "parizer"), where laws are just as strict as in France.

Anyhow, here's what I get in France if I remove the Economist cookie (in a separate frame overlayed on the Economist website): http://consent-pref.truste.com/?type...ste.com/notice

I know Python, so I gave it a shot. But I know nothing about Mechanize or automatically navigating the web with Python (and no time/desire to learn it), so I failed.

Basically, you need to automatically "click" the "agree and proceed" button, which is somehow generated by javascript on the link I included above (I guess... I really don't have a clue about website design/generation, sorry). I tried to do this by adding a javascript_login(...) function with some form and submit stuff in the Economist recipe, but I gave up. Maybe that isn't the right approach anyway.

I hope my aborted attempt provides a good starting point for people who know more about Mechanize and javascript.

[kovidgoyal]

I doubt it's the cookie acceptance. France is the only country inEuriope where it has been reported, and the cookie acceptance is usually implemented in javascript, so it would have no affect on mechanize, which does not execute javascript.

[NSILMike]

fstulp-
I'm in the US and it works for me as I've noted above. However, I've never been to the economist website, so would not have gotten a cookie or consented to their use, etc. except what Calibre could do. However, I also use a windows app that cleans all cookies and tmp files at boot, so no economist cookie could persist... Thus your hypothesis might be correct, if the site is detecting your origin.

[Divingduck]

I did a check from germay. works for me too. I accept the cookie after the download half hour ago this morning. So, for me it seems to work without cookie too.

[MoiMoiMoi]

Hello,

It is indeed related to the cookie acceptance notice.
Using wget to get the page
http//www.economist.com/printedition without any cookie gives a page that contains:

<title>Explicit cookie consent | The Economist</title>


However, the same page also contains the following javascript code:

Function checkForAcceptance() {
if (document.cookie.indexOf('notice_preferences') !== -1) {
if (window.location.href.indexOf('explicit-cookie-consent.html') === -1) {
window.location.reload();
}
else {
window.location = '/';
window.location.replace('/');
}
}
else {
setTimeout(checkForAcceptance, 100);
}
}
// Start the watch off.
checkForAcceptance();


So... setting the "notice_preferences" cookie should do the trick.

Starting from:
https://github.com/kovidgoyal/calibr...st_free.recipe

1. Uncomment the lines 61-75 (the get_browser function)
2. Add the following lines after line 63:

import cookielib
ck = cookielib.Cookie(version=0, name='notice_preferences', value='2:',
port=None, port_specified=False, domain='.economist.com',
domain_specified=False, domain_initial_dot=True,
path='/', path_specified=False, secure=False,
expires=None, discard=False, comment=None,
comment_url=None, rest={'HttpOnly': None}, rfc2109=False)
br.cookiejar.set_cookie(ck)

This make the recipe work for me (In France, calibre 2.3.0, Linux).

I am not posting this as a patch as I don't know nearly enough about cookies, mechanize and calibre to make it a proper patch. However, since the calibre author has seen this thread, maybe he can provide some advice.

Cheers.

[kovidgoyal]

Your patch looks fine, I will add it to the next release.

[Parisien]

Many thanks indeed, MoiMoiMoi. What's strange (for me anyhow, and I don't understand a word of coding) is that it downloaded automatically to my French IP this morning, and I'm still on Calibre 2.3!

[fstulp]

Originally Posted by Parisien

What's strange (for me anyhow, and I don't understand a word of coding) is that it downloaded automatically to my French IP this morning, and I'm still on Calibre 2.3!

Strange indeed. As I suspected (see my previous post) and as MoiMoiMoi cleverly confirmed with wget, the cookie acceptance was the problem. But...
* The cookie still needs to be confirmed on the Economist website in France
* The MoiMoiMoi patch has not yet been applied (at least on my calibre version: 2.7)
* I can also download the Economist from a French IP again!

I don't understand how this could work again, apart from some change in cookie acceptance on the website...

Anyhow, it works again: Happy reading!