Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Recipes

Notices

Reply
 
Thread Tools Search this Thread
Old 08-04-2019, 10:47 AM   #1
bobbysteel
Big Poppa
bobbysteel began at the beginning.
 
Posts: 103
Karma: 10
Join Date: Jul 2010
Device: Nook
Times UK Failing right at recipe Download

On multiple machines it gets to either
Downloading recipe urn: custom:1005 on the windows app or
Using downloaded builtin recipe on command line linux.
and freezes.

windows shows an error
Code:
Python function terminated unexpectedly
  <urlopen error [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond> (Error Code: 1)
Traceback (most recent call last):
  File "site.py", line 101, in main
  File "site.py", line 78, in run_entry_point
  File "site-packages\calibre\utils\ipc\worker.py", line 199, in main
  File "site-packages\calibre\gui2\convert\gui_conversion.py", line 35, in gui_convert_recipe
  File "site-packages\calibre\gui2\convert\gui_conversion.py", line 27, in gui_convert
  File "site-packages\calibre\ebooks\conversion\plumber.py", line 1106, in run
  File "site-packages\calibre\customize\conversion.py", line 244, in __call__
  File "site-packages\calibre\ebooks\conversion\plugins\recipe_input.py", line 135, in convert
  File "site-packages\calibre\web\feeds\news.py", line 898, in __init__
  File "<string>", line 86, in get_browser
  File "site-packages\mechanize\_mechanize.py", line 254, in open
  File "site-packages\mechanize\_mechanize.py", line 284, in _mech_open
  File "site-packages\mechanize\_opener.py", line 195, in open
  File "site-packages\mechanize\_urllib2_fork.py", line 352, in _open
  File "site-packages\mechanize\_urllib2_fork.py", line 340, in _call_chain
  File "site-packages\calibre\utils\browser.py", line 28, in https_open
  File "site-packages\mechanize\_urllib2_fork.py", line 1160, in do_open
urllib2.URLError: <urlopen error [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>
but all the URLs in the recipe resolve properly. also other papers download fine. v odd. any better way to debug I can try?
bobbysteel is offline   Reply With Quote
Old 08-04-2019, 11:35 AM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,093
Karma: 15014853
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
That's an error originating at line 86 in the recipe. Not a failure to download the recipe
kovidgoyal is offline   Reply With Quote
Advert
Old 08-04-2019, 01:50 PM   #3
bobbysteel
Big Poppa
bobbysteel began at the beginning.
 
Posts: 103
Karma: 10
Join Date: Jul 2010
Device: Nook
Weird. Kovid that is a simple login line to https://login.thetimes.co.uk/
Could you advise how to further troubleshoot? Curling that URL works fine for me.
bobbysteel is offline   Reply With Quote
Old 08-04-2019, 02:12 PM   #4
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,093
Karma: 15014853
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
most likely something is blocking calibre those often manifest as timeouts
kovidgoyal is offline   Reply With Quote
Old 08-04-2019, 03:57 PM   #5
bobbysteel
Big Poppa
bobbysteel began at the beginning.
 
Posts: 103
Karma: 10
Join Date: Jul 2010
Device: Nook
Quote:
Originally Posted by kovidgoyal View Post
most likely something is blocking calibre those often manifest as timeouts
Is there an easier way I can debug? I assumed curl would show any connection issues. Do you get the same error? I can pm you a login
bobbysteel is offline   Reply With Quote
Advert
Old 08-05-2019, 01:12 AM   #6
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,093
Karma: 15014853
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
I get a hang on line 89 not line 86 where the recipe is doing a POST. And you can use

br.set_debug_http(True)


to get more detailed output about what is happening.
kovidgoyal is offline   Reply With Quote
Old 08-05-2019, 02:35 PM   #7
bobbysteel
Big Poppa
bobbysteel began at the beginning.
 
Posts: 103
Karma: 10
Join Date: Jul 2010
Device: Nook
Code:
    def get_browser(self):
        USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0'
        br = BasicNewsRecipe.get_browser(self, user_agent=USER_AGENT)
        #br.set_debug_http(True)
        #br.set_debug_responses(True)
        br.set_debug_http(True)
        br.open(self.INDEX)
        if self.username is not None and self.password is not None:
            #data = {'login': self.username, 'password':self.password}
            data = dict(login=self.username,password=self.password)
            br.open(self.LOGIN, data)
            response = br.response()
        return br
I tried this - it seems from the debug it's a problem of submitting as a GET not as a POST. But trying to make a dict fails - I get unhashableType errors. Probably a stupid mistake?
bobbysteel is offline   Reply With Quote
Old 08-05-2019, 03:15 PM   #8
bobbysteel
Big Poppa
bobbysteel began at the beginning.
 
Posts: 103
Karma: 10
Join Date: Jul 2010
Device: Nook
Weird, I can get it to post but still get error
Code:
Python function terminated unexpectedly
  <urlopen error [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond> (Error Code: 1)
Traceback (most recent call last):
  File "site.py", line 101, in main
  File "site.py", line 78, in run_entry_point
  File "site-packages\calibre\utils\ipc\worker.py", line 199, in main
  File "site-packages\calibre\gui2\convert\gui_conversion.py", line 35, in gui_convert_recipe
  File "site-packages\calibre\gui2\convert\gui_conversion.py", line 27, in gui_convert
  File "site-packages\calibre\ebooks\conversion\plumber.py", line 1106, in run
  File "site-packages\calibre\customize\conversion.py", line 244, in __call__
  File "site-packages\calibre\ebooks\conversion\plugins\recipe_input.py", line 135, in convert
  File "site-packages\calibre\web\feeds\news.py", line 898, in __init__
  File "<string>", line 66, in get_browser
  File "site-packages\mechanize\_mechanize.py", line 254, in open
  File "site-packages\mechanize\_mechanize.py", line 284, in _mech_open
  File "site-packages\mechanize\_opener.py", line 195, in open
  File "site-packages\mechanize\_urllib2_fork.py", line 352, in _open
  File "site-packages\mechanize\_urllib2_fork.py", line 340, in _call_chain
  File "site-packages\calibre\utils\browser.py", line 28, in https_open
  File "site-packages\mechanize\_urllib2_fork.py", line 1160, in do_open
urllib2.URLError: <urlopen error [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>
Debug is unhelpful. Command line curl works great. Any ideas how to get further debug data?
bobbysteel is offline   Reply With Quote
Old 08-05-2019, 03:16 PM   #9
bobbysteel
Big Poppa
bobbysteel began at the beginning.
 
Posts: 103
Karma: 10
Join Date: Jul 2010
Device: Nook
Code:
Sending login request...
send: 'POST / HTTP/1.1\r\nAccept-Encoding: identity\r\nAccept-Language: en-US,en;q=0.8\r\nConnection: close\r\nAccept: text/html\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0\r\nCookie: nuk_customer_time_zone=GMT; nuk_customer_country_code=GB; nuk_customer_location_hint=UK\r\nContent-Length: 40\r\nContent-Type: application/x-www-form-urlencoded\r\nHost: login.thetimes.co.uk\r\nX-Http-Method-Override: POST\r\n\r\nlogin=times%40xxxxx.com&password=xxxxxxx'
bobbysteel is offline   Reply With Quote
Old 08-05-2019, 09:38 PM   #10
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,093
Karma: 15014853
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
The POST is working with curl and not mechanize?
kovidgoyal is offline   Reply With Quote
Old 08-06-2019, 10:46 AM   #11
bobbysteel
Big Poppa
bobbysteel began at the beginning.
 
Posts: 103
Karma: 10
Join Date: Jul 2010
Device: Nook
Quote:
Originally Posted by kovidgoyal View Post
The POST is working with curl and not mechanize?
Strange. I tested just the GET of the page itself, but it's hanging on the POST itself. The manual CURL redirects via 302 then throws up an error with Akamai
Code:
< HTTP/1.1 411 Length Required
< Server: AkamaiGHost
With curl I tried
Code:
-d "Content-Length: 108"
but that doesn't work either. Setting Calibre to allow redirects doesn't change the error. Are there other recipes that I should refer to for situations like this? It seems like quite a standard login box sadly.
bobbysteel is offline   Reply With Quote
Old 08-06-2019, 10:56 AM   #12
bobbysteel
Big Poppa
bobbysteel began at the beginning.
 
Posts: 103
Karma: 10
Join Date: Jul 2010
Device: Nook
fwiw the curl returns the 302 URL that has a valid token in it. the browser needs to return that or follow it directly to then authorize the session...
bobbysteel is offline   Reply With Quote
Old 08-06-2019, 11:07 AM   #13
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,093
Karma: 15014853
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
turn off following redirects in the browser and manually get the token from the response headers in that case.
kovidgoyal is offline   Reply With Quote
Old 08-06-2019, 11:13 AM   #14
bobbysteel
Big Poppa
bobbysteel began at the beginning.
 
Posts: 103
Karma: 10
Join Date: Jul 2010
Device: Nook
Quote:
Originally Posted by kovidgoyal View Post
turn off following redirects in the browser and manually get the token from the response headers in that case.
how can i get the reponse headers? the POST doesn't return properly in Calibre. in command line curl I can at least get back the 302 destination. in my test recipe now i have
Code:
            data = urlencode({
                'gotoUrl': self.INDEX,
                's': 1,
                'username': self.username,
                'password': self.password})
            geturl = br.open('https://login.thetimes.co.uk', data).geturl()
but the open never resolves as it hangs.
bobbysteel is offline   Reply With Quote
Old 08-06-2019, 11:17 AM   #15
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,093
Karma: 15014853
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
turn off following redirects and it wont hang. IIRC set_handle_redirect(False)
kovidgoyal is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
New Yorker recipe failing adfadfsasdfafafd Recipes 2 03-28-2017 03:47 PM
"The New York Times" recipe failing with error mikebw Recipes 8 10-02-2015 06:48 PM
New York Times Recipe failing to verify SSL Cert Ramblurr Recipes 6 02-27-2015 05:31 AM
download of wsj failing amritsari Calibre 4 09-06-2012 06:53 PM
Economist Recipe Failing... awitko Recipes 2 11-07-2011 12:47 AM


All times are GMT -4. The time now is 11:04 AM.


MobileRead.com is a privately owned, operated and funded community.