Issues with recipe login

[jfhutson]

Would anyone be willing to help me with my recipe for First Things magazine (www.firstthings.com)?

My code is here: https://github.com/jfhutson/calibre-...tthings.recipe

It seems to be working except that it won't log in, even when I give it my login information, so I get nothing from the articles behind the paywall.

[kovidgoyal]

general login debuggin tips:

inspect the html your get back from br.submit() like this
html = br.submit().read()

see if it contains anything indicating the login failed or succeeded.

Do the login in chrome with the developer tools enabled to see the exact request chain that is sent to the server. You can then duplicate that in the recipe using mechanize.Request, see for example the builtin wsj recipe

[jfhutson]

Thanks!
The response I get from br.response.read() is:

<response_seek_wrapper at 0x4617c08L whose wrapped object = <closeable_response at 0x4936988L whose fp = <socket._fileobject object at 0x000000000492F1B0>>>

I think that might represent a thing that comes up when you log in saying your login was successful or unsuccessful, but I can't tell which. When I do br.submit().read() I can tell that I have not been logged in.

I tried replicating the WSJ recipe, but was still unsuccessful. (https://github.com/jfhutson/calibre-...tthings.recipe)

For what it's worth here are the request headers when I log in:

Request URL:https://www.firstthings.com/ajax/login
Request Method:POST
Status Code:200 OK
Remote Address:67.225.137.221:443
Response Headers
view source
Cache-Control:no-store, no-cache, must-revalidate
Connection:Keep-Alive, Keep-Alive
Content-Length:54
Content-Type:text/html; charset=UTF-8
Date:Sun, 19 Feb 2017 21:28:16 GMT
Expires:Thu, 19 Nov 1981 08:52:00 GMT
Keep-Alive:timeout=5, max=100
Pragma:no-cache
Server:Apache
Set-Cookie:subLogin=fa2dd5c1-6799-488a-8006-e4898fd77d26; expires=Sun, 26-Feb-2017 21:28:17 GMT; Max-Age=604800; path=/; HttpOnly
X-UA-Compatible:IE=edge
Request Headers
view source
Accept:application/json, text/javascript, */*; q=0.01
Accept-Encoding:gzip, deflate, br
Accept-Language:en-US,en;q=0.8
Connection:keep-alive
Content-Length:47
Content-Type:application/x-www-form-urlencoded; charset=UTF-8
Cookie:__qca=P0-1778486508-1485632321642; __gads=ID=4ef5141946d2bbc1:T=1485632322:S=ALNI_MZi U_KFqXZF-fpYmjLTqHphsbCcCA; __utma=40115125.517541984.1485632321.1485632321.14 85632321.1; __utmz=40115125.1485632321.1.1.utmcsr=google|utmcc n=(organic)|utmcmd=organic|utmctr=(not%20provided) ; __ar_v4=TNWIWF2655BFZLSZBIE2LG%3A20170127%3A2%7C6U L2ZQMXT5FB7JUSY3B2ZM%3A20170127%3A2%7C6QCBJSM7EBCJ ZAR673BHPB%3A20170127%3A2; PHPSESSID=dv3j5ak9ln4ortinjbq20embh0
DNT:1
Host:www.firstthings.com
Origin:https://www.firstthings.com
Referer:https://www.firstthings.com/current-edition
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36
X-Requested-With:XMLHttpRequest
Form Data
view source
view URL encoded
username:[redacted]
password:[redacted]

[kovidgoyal]

Use set_debug_http(True) on the mechanize browser to see the exact requests being sent and compare. It's likely that there is some javascript on the page that sets some extra needed cookies which you will have to do manually in the recipe.

[gcf]

Using the Chrome dev tools, it looks like the FirstThings website sets a cookie named "subLogin" set to a value which looks like a v4 UUID. Deleting the "subLogin" cookie and refreshing results in the site no longer considering me logged in.

When I modify jfhutson's github recipe to call set_debug_http(True) on the mechanize browser, the output does not include "subLogin."

I think the "subLogin" cookie is the extra cookie Kovid was referring to.

(I recognize the thread is old, but I'm trying to work on this same problem.)

[kovidgoyal]

If it is just a random cookie its very easy to set in the recipe

from uuid import uuid4
br.set_cookie('subLogin', str(uuid4()), '.whatever-domain.com'))