Times UK Failing right at recipe Download

[bobbysteel]

On multiple machines it gets to either
Downloading recipe urn: custom:1005 on the windows app or
Using downloaded builtin recipe on command line linux.
and freezes.

windows shows an error

Python function terminated unexpectedly
  <urlopen error [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond> (Error Code: 1)
Traceback (most recent call last):
  File "site.py", line 101, in main
  File "site.py", line 78, in run_entry_point
  File "site-packages\calibre\utils\ipc\worker.py", line 199, in main
  File "site-packages\calibre\gui2\convert\gui_conversion.py", line 35, in gui_convert_recipe
  File "site-packages\calibre\gui2\convert\gui_conversion.py", line 27, in gui_convert
  File "site-packages\calibre\ebooks\conversion\plumber.py", line 1106, in run
  File "site-packages\calibre\customize\conversion.py", line 244, in __call__
  File "site-packages\calibre\ebooks\conversion\plugins\recipe_input.py", line 135, in convert
  File "site-packages\calibre\web\feeds\news.py", line 898, in __init__
  File "<string>", line 86, in get_browser
  File "site-packages\mechanize\_mechanize.py", line 254, in open
  File "site-packages\mechanize\_mechanize.py", line 284, in _mech_open
  File "site-packages\mechanize\_opener.py", line 195, in open
  File "site-packages\mechanize\_urllib2_fork.py", line 352, in _open
  File "site-packages\mechanize\_urllib2_fork.py", line 340, in _call_chain
  File "site-packages\calibre\utils\browser.py", line 28, in https_open
  File "site-packages\mechanize\_urllib2_fork.py", line 1160, in do_open
urllib2.URLError: <urlopen error [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>

but all the URLs in the recipe resolve properly. also other papers download fine. v odd. any better way to debug I can try?

[kovidgoyal]

That's an error originating at line 86 in the recipe. Not a failure to download the recipe

[bobbysteel]

Weird. Kovid that is a simple login line to https://login.thetimes.co.uk/
Could you advise how to further troubleshoot? Curling that URL works fine for me.

[kovidgoyal]

most likely something is blocking calibre those often manifest as timeouts

[bobbysteel]

Originally Posted by kovidgoyal

most likely something is blocking calibre those often manifest as timeouts

Is there an easier way I can debug? I assumed curl would show any connection issues. Do you get the same error? I can pm you a login

[kovidgoyal]

I get a hang on line 89 not line 86 where the recipe is doing a POST. And you can use

br.set_debug_http(True)


to get more detailed output about what is happening.

[bobbysteel]

    def get_browser(self):
        USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0'
        br = BasicNewsRecipe.get_browser(self, user_agent=USER_AGENT)
        #br.set_debug_http(True)
        #br.set_debug_responses(True)
        br.set_debug_http(True)
        br.open(self.INDEX)
        if self.username is not None and self.password is not None:
            #data = {'login': self.username, 'password':self.password}
            data = dict(login=self.username,password=self.password)
            br.open(self.LOGIN, data)
            response = br.response()
        return br

I tried this - it seems from the debug it's a problem of submitting as a GET not as a POST. But trying to make a dict fails - I get unhashableType errors. Probably a stupid mistake?

[bobbysteel]

Weird, I can get it to post but still get error

Python function terminated unexpectedly
  <urlopen error [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond> (Error Code: 1)
Traceback (most recent call last):
  File "site.py", line 101, in main
  File "site.py", line 78, in run_entry_point
  File "site-packages\calibre\utils\ipc\worker.py", line 199, in main
  File "site-packages\calibre\gui2\convert\gui_conversion.py", line 35, in gui_convert_recipe
  File "site-packages\calibre\gui2\convert\gui_conversion.py", line 27, in gui_convert
  File "site-packages\calibre\ebooks\conversion\plumber.py", line 1106, in run
  File "site-packages\calibre\customize\conversion.py", line 244, in __call__
  File "site-packages\calibre\ebooks\conversion\plugins\recipe_input.py", line 135, in convert
  File "site-packages\calibre\web\feeds\news.py", line 898, in __init__
  File "<string>", line 66, in get_browser
  File "site-packages\mechanize\_mechanize.py", line 254, in open
  File "site-packages\mechanize\_mechanize.py", line 284, in _mech_open
  File "site-packages\mechanize\_opener.py", line 195, in open
  File "site-packages\mechanize\_urllib2_fork.py", line 352, in _open
  File "site-packages\mechanize\_urllib2_fork.py", line 340, in _call_chain
  File "site-packages\calibre\utils\browser.py", line 28, in https_open
  File "site-packages\mechanize\_urllib2_fork.py", line 1160, in do_open
urllib2.URLError: <urlopen error [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>

Debug is unhelpful. Command line curl works great. Any ideas how to get further debug data?

[bobbysteel]

Sending login request...
send: 'POST / HTTP/1.1\r\nAccept-Encoding: identity\r\nAccept-Language: en-US,en;q=0.8\r\nConnection: close\r\nAccept: text/html\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0\r\nCookie: nuk_customer_time_zone=GMT; nuk_customer_country_code=GB; nuk_customer_location_hint=UK\r\nContent-Length: 40\r\nContent-Type: application/x-www-form-urlencoded\r\nHost: login.thetimes.co.uk\r\nX-Http-Method-Override: POST\r\n\r\nlogin=times%40xxxxx.com&password=xxxxxxx'

[kovidgoyal]

The POST is working with curl and not mechanize?

[bobbysteel]

Originally Posted by kovidgoyal

The POST is working with curl and not mechanize?

Strange. I tested just the GET of the page itself, but it's hanging on the POST itself. The manual CURL redirects via 302 then throws up an error with Akamai

< HTTP/1.1 411 Length Required
< Server: AkamaiGHost

With curl I tried

-d "Content-Length: 108"

but that doesn't work either. Setting Calibre to allow redirects doesn't change the error. Are there other recipes that I should refer to for situations like this? It seems like quite a standard login box sadly.

[bobbysteel]

fwiw the curl returns the 302 URL that has a valid token in it. the browser needs to return that or follow it directly to then authorize the session...

[kovidgoyal]

turn off following redirects in the browser and manually get the token from the response headers in that case.

[bobbysteel]

Originally Posted by kovidgoyal

turn off following redirects in the browser and manually get the token from the response headers in that case.

how can i get the reponse headers? the POST doesn't return properly in Calibre. in command line curl I can at least get back the 302 destination. in my test recipe now i have

            data = urlencode({
                'gotoUrl': self.INDEX,
                's': 1,
                'username': self.username,
                'password': self.password})
            geturl = br.open('https://login.thetimes.co.uk', data).geturl()

but the open never resolves as it hangs.

[kovidgoyal]

turn off following redirects and it wont hang. IIRC set_handle_redirect(False)