Times UK Failing right at recipe Download

[bobbysteel]

this doesn't work https://stackoverflow.com/questions/...thon-mechanize

[kovidgoyal]

Since calibre 4 is not released yet, there are no recipes that do that, but if you are asking how to set cookies on a mechanize broser simply call either set_cookie() with the string from the Set-Cookie header or alternately set_simple_cookie() with the name, domain and path of the cookie.

[bobbysteel]

would a mechanize cookie jar work? I'm trying this now and dumping my full cookies.txt but it's not authing.

[kovidgoyal]

If set_cookie is working then a cookie jar is not going to work either.

[bobbysteel]

Originally Posted by kovidgoyal

If set_cookie is working then a cookie jar is not going to work either.

i can't try set cookie as there are over a dozen fields and i don't have the hours to rewrite each into set cookie format. the cookies.txt should work but i don't know how to even do a log of it in Python to fix this. in JS I'd juse JSON.stringify but when ido a self.log on the cookie jar it just shows object of type MozillaCookieJar but can't even tell the contents are being loaded

[kovidgoyal]

If the cookies are being loaded, then they would be present as Cookie headers when you use set_debug_http(). If you want to check if a jar has cookies, simply use save() to save it to a file and inspect that.

[bobbysteel]

I can see SOME of the cookies in the debug but they're not all getting passed through

       kw['user_agent'] = random_user_agent(allow_ie=False)
        br = BasicNewsRecipe.get_browser(self, *a, **kw)
        self.log('Starting login process...')
        br.set_debug_redirects(True)
        br.set_debug_http(True)
        cookiejar = mechanize.FileCookieJar('/home/me/calibre-commandline/cookies.txt')
        #br.set_handle_robots(False)
        br.set_cookiejar(cookiejar)                                                                                             #self.log(cookiejar)
        res = br.open(start_url)

this fails to login. but just curl'ing with curl -c cookies.txt <paywalled url> works fine.

looking at the debug headers some of the cookie headers are being passed but not all of them. the cookie file looks fine though.

[kovidgoyal]

Whether a cookie gets picked up or not depends on its path, expiration date, domain, etc. It should be fairly easy to figure out which is th eimportant cookie by doing a binary search on cookies.txt with curl and then manually setting that one using set_simple_cookie() which should avoid issues with expiration dates, etc.

[bobbysteel]

Hm. So with curl I've isolated it to two cookies acs_tnl and sacs_tnl
Manually calling with curl works fine reading the cookies.txt with just those.
However in the recipe

        br.set_debug_redirects(True)
        br.set_debug_http(True)
        br.addheaders = [('Cookie','acs_tnl=xxxxxxxxx')]       
        br.addheaders = [('Cookie','sacs_tnl=f')]

still fails.
Curl with just those two cookies also seems to fail. I can't seem to figure out why the same text file works as a file but not manually.

[kovidgoyal]

Hmm well if it is not working with curl with those two cookies, it's not likely to work with mechanize either.

[bobbysteel]

ah i've realized there's also some antisharing mechanism now kicking in

[bobbysteel]

Originally Posted by kovidgoyal

Hmm well if it is not working with curl with those two cookies, it's not likely to work with mechanize either.

it's weird - the auth0 system is rather complex. I've tried using headless chrome to generate a cookie file and then using that in the recipe, but won't work. for your v4 recipes, how will they differ in cases like this? i'm running your 3.99 beta and would like to play around with it.
cheers

[kovidgoyal]

The recipes will be no different. You simply have the option of running a headless browser to extract cookies. And eventually I may add some convenience APIs to download the entire content using a headless browser. No need to use mechanize at all.

[bobbysteel]

Originally Posted by kovidgoyal

The recipes will be no different. You simply have the option of running a headless browser to extract cookies. And eventually I may add some convenience APIs to download the entire content using a headless browser. No need to use mechanize at all.

truly weird. with headless chrome i can automate the login no problem, but mechanize doesn't seem to work even with the same cookies set post-login. any ideas what else would differ between the two tools?

[kovidgoyal]

javascript can set arbitrary headers, not just cookies. So you have to resort to exaing the actual HTTP requests to see the difference.