Irish Times - Problems Entering Subscription

[leo738]

I notice in the WSJ a cookie is set:

br.set_cookie('m', data['username'], '.wsj.com')

Do I need to do the same? I've looked at the cookies & I can see one marked:

IT_PW_AUTH:"1481573584.8521186.2645c2a989c9003b473 0bbbdf4dac8b4.a4b3aef76e90c9aeda5a4f13ee0b770b9ea9 41c39c63ba57429b741a422ffba9"

Perhaps this is for paywall authorisation? How would I set this in the recipe?

[kovidgoyal]

If you want to determine what cookies are needed you can do so by deleting cookies from the browser cookie store and seeing when the login stops working. As for how to set cookies, the builtin wsj recipe has an example of doing that.

[leo738]

Thanks but my issue is trying to understand the process of setting the cookies. On sigin the server responds with the following:

{u'error_number': u'0', u'firstname': u'Leo', u'session_token': u'4530a73ca859d9c6b89eb7cfc2b5c17d', u'user_id': u'8527987', u'error_message': u'', u'entitlements': [u'app', u'crossword', u'archive', u'paywall', u'epaper'], u'varnish_id': u'1481665264.8521186.4530a73ca859d9c6b89eb7cfc2b5c17d.b53fed675635e3e1f8877a48ceb7436f7fe183c6d50532c1dcb173a707132a21'}

I'm not sure how I should set this using

br.set_cookie(........)

Any pointers??

[kovidgoyal]

I dont understand what you are asking. There is an axample of setting cookies in the wsj recipe, you call br.set_cookie(name, data, domain) To find out what name and data should be, look in your browsers cookie store, to see what cookies get set by the login process.

[leo738]

Thanks for the reply. A bit confusing though. When I look in my browser after login I see cookies such as:

IT_paywall
IT_PW_AUTH
IT_UUID

etc. etc.

(I notice some have domain name associated with them, some domain. Don't know what the difference is)

but on the command line response from the server to login I see something like:

{u'error_number': u'0', u'firstname': u'Leo', u'varnish_id': u'1481723438.8521186.4e0191ae22e2f018d26a62b41b064388.1ad0aa2e097b79b4e54b7951b7e73b1ccc7654cd840e55cd4604c9263d8590cf', u'error_message': u'', u'entitlements': [u'app', u'crossword', u'archive', u'paywall', u'epaper'], u'session_token': u'4e0191ae22e2f018d26a62b41b064388', u'user_id': u'8515186'}

So I was unsure of what name to use. I'm going to try guessing at what corresponds to what & have a go, e.g:

br.set_cookie('IT_entitlements', data['entitlements'], '.irishtimes.com')
        br.set_cookie('IT_pw_userdata', data['user_id'], '.irishtimes.com')         
        br.set_cookie('IT_paywall', data['session_token'], '.irishtimes.com')
        br.set_cookie('IT_PW_AUTH', data['varnish_id'], '.irishtimes.com')

[kovidgoyal]

Where are you getting

{u'error_number': u'0', u'firstname': u'Leo', u'varnish_id': u'1481723438.8521186.4e0191ae22e2f018d26a62b41b064 388.1ad0aa2e097b79b4e54b7951b7e73b1ccc7654cd840e55 cd4604c9263d8590cf', u'error_message': u'', u'entitlements': [u'app', u'crossword', u'archive', u'paywall', u'epaper'], u'session_token': u'4e0191ae22e2f018d26a62b41b064388', u'user_id': u'8515186'}

from? Cookies are set via Set-Cookie headers in normal HTTP. With javascript, the javascript code can set any data it likes with any amount of processing in between. So you need to match up whatever data you can find in the servers responses with the cookies you can see in the browser. The cookie names may or may not be realted to what you see in the server response, you have to match by guessing using the name and the value. Or just pretty print the javascript from the page and trace the executaion, that is sometimes easier, depending on how good you are at reading javascript.

[leo738]

The string was from the server after login by outputting the server response using:

r = br.open(rq)
raw = r.read()
data = json.loads(raw)
print(data)

It seems I got lucky on my first stab at the setting the cookie!

br.set_cookie('IT_PW_AUTH', data['varnish_id'], '.irishtimes.com')

appears to result in a fully working recipe, attached below.

Regards,

Leo

[kovidgoyal]

Congratulations

You might want to randomize the device_id, like this

deviceid = str(uuid4()).replace('-', '')

[leo738]

Many thanks for the suggestion, I tried without a device ID & it failed so seems to be crucial. Just corrected the syntax:

deviceid = str(uuid.uuid4()).replace('-', '')

Updated the recipe, latest attached.

Leo

[Cerdito]

Hi,

When I run this with the line:

br.set_cookie('IT_PW_AUTH', data['varnish_id'], '.irishtimes.com')

I get the following error when I run it from the command line:

TypeError: set_cookie() takes exactly 2 arguments (4 given)

Any idea what I am doing wrong?

[kovidgoyal]

You are using an out of date version of calibre. The correct form of set_cookie was implemented about a year ago.

[Cerdito]

Thanks, you were right, I was running Calibre 0.8 on Raspbian.

I'm getting an error when I try to upgrade using the script from the Calibre website ("OSError: [Errno 8] Exec format error"), what am I doing wrong here? Output below



pi@raspberrypi /opt/calibre $ sudo -v && wget -nv -O- https://download.calibre-ebook.com/linux-installer.py | sudo python -c "import sys; main=lambda:sys.stderr.write('Download failed\n'); exec(sys.stdin.read()); main()"
[sudo] password for pi:
2017-03-02 15:09:43 URL:https://download.calibre-ebook.com/linux-installer.py [27407/27407] -> "-" [1]
Installing to /opt/calibre
Downloading tarball signature securely...
Using previously downloaded calibre-2.80.0-i686.txz
Extracting files to /opt/calibre ...
Extracting application files...
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "<string>", line 726, in main
File "<string>", line 687, in run_installer
File "/usr/lib/python2.7/subprocess.py", line 493, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1259, in _execute_child
raise child_exception
OSError: [Errno 8] Exec format error

[PeterT]

I believe for raspberry pi you need to download and build from source. If you check what was downloaded it was an Intel binary (i686.txz)