Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Recipes

Notices

Reply
 
Thread Tools Search this Thread
Old 08-06-2019, 11:20 AM   #16
bobbysteel
Big Poppa
bobbysteel began at the beginning.
 
Posts: 103
Karma: 10
Join Date: Jul 2010
Device: Nook
Tried that now
Code:
Python function terminated unexpectedly
  HTTP Error 301: Moved Permanently (Error Code: 1)
Traceback (most recent call last):
  File "site.py", line 101, in main
  File "site.py", line 78, in run_entry_point
  File "site-packages\calibre\utils\ipc\worker.py", line 200, in main
  File "site-packages\calibre\gui2\convert\gui_conversion.py", line 35, in gui_convert_recipe
  File "site-packages\calibre\gui2\convert\gui_conversion.py", line 27, in gui_convert
  File "site-packages\calibre\ebooks\conversion\plumber.py", line 1110, in run
  File "site-packages\calibre\customize\conversion.py", line 246, in __call__
  File "site-packages\calibre\ebooks\conversion\plugins\recipe_input.py", line 138, in convert
  File "site-packages\calibre\web\feeds\news.py", line 898, in __init__
  File "<string>", line 55, in get_browser
  File "site-packages\mechanize\_mechanize.py", line 254, in open
  File "site-packages\mechanize\_mechanize.py", line 310, in _mech_open
mechanize._response.httperror_seek_wrapper: HTTP Error 301: Moved Permanently
Fails but differently
bobbysteel is offline   Reply With Quote
Old 08-06-2019, 11:25 AM   #17
bobbysteel
Big Poppa
bobbysteel began at the beginning.
 
Posts: 103
Karma: 10
Join Date: Jul 2010
Device: Nook
duplicate
bobbysteel is offline   Reply With Quote
Advert
Old 08-06-2019, 11:41 AM   #18
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,093
Karma: 15014853
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
catch the error, it should have the information about the redirect.
kovidgoyal is offline   Reply With Quote
Old 08-06-2019, 11:49 AM   #19
bobbysteel
Big Poppa
bobbysteel began at the beginning.
 
Posts: 103
Karma: 10
Join Date: Jul 2010
Device: Nook
I stole the WSJ code and it seems to work again with that hacked down. Submitting PR now
bobbysteel is offline   Reply With Quote
Old 08-30-2019, 09:25 AM   #20
bobbysteel
Big Poppa
bobbysteel began at the beginning.
 
Posts: 103
Karma: 10
Join Date: Jul 2010
Device: Nook
Kovid - it looks like they made further changes where https://account.thetimes.co.uk/login no longer works as it now has a state variable that comes from the redirect at login.thetimes.co.uk. This looks similar to the WSJ SSO format but not quite. Do you recognize this style from other recipes per chance?
bobbysteel is offline   Reply With Quote
Advert
Old 08-30-2019, 10:38 AM   #21
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,093
Karma: 15014853
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
I had to recently update the WSJ recipe, maybe you can try those changes.
kovidgoyal is offline   Reply With Quote
Old 08-31-2019, 01:42 PM   #22
bobbysteel
Big Poppa
bobbysteel began at the beginning.
 
Posts: 103
Karma: 10
Join Date: Jul 2010
Device: Nook
the Times doesn't seem to use quite the same javascript - it's weird. does it look familar at all to you? it's a sso type login but not the same variables used at all for example
bobbysteel is offline   Reply With Quote
Old 08-31-2019, 02:11 PM   #23
bobbysteel
Big Poppa
bobbysteel began at the beginning.
 
Posts: 103
Karma: 10
Join Date: Jul 2010
Device: Nook
I've tried updating all the variables to match but it hangs on the initialization


https://login.thetimes.co.uk/
send: 'GET / HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: login.thetimes.co.uk\r\nConnection: close\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.75 Safari/537.36\r\nAccept: */*\r\n\r\n'
bobbysteel is offline   Reply With Quote
Old 08-31-2019, 02:14 PM   #24
bobbysteel
Big Poppa
bobbysteel began at the beginning.
 
Posts: 103
Karma: 10
Join Date: Jul 2010
Device: Nook
it's odd the website functions fine albeit with a bunch of forwards when you inspect in chrome
bobbysteel is offline   Reply With Quote
Old 08-31-2019, 03:11 PM   #25
bobbysteel
Big Poppa
bobbysteel began at the beginning.
 
Posts: 103
Karma: 10
Join Date: Jul 2010
Device: Nook
Code:
  if needs_subscription:
        def get_browser(self, *a, **kw):
            # To understand the login logic read app-min.js from
            # https://sso.accounts.dowjones.com/login
            itp = quote(self.INDEX, safe='')
            start_url = 'https://login.thetimes.co.uk/'
            kw['user_agent'] = random_user_agent(allow_ie=False)
            br = BasicNewsRecipe.get_browser(self, *a, **kw)
            br.set_debug_http(True)
            self.log('Starting login process...')
            self.log(start_url)
            res = br.open(start_url)
            sso_url = res.geturl()
            self.log('Get sso URL')
            self.log(sso_url)
            query =  urlparse.parse_qs(urlparse.urlparse(sso_url).query)
this fails before it gets to the sso URL log line
bobbysteel is offline   Reply With Quote
Old 08-31-2019, 08:59 PM   #26
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,093
Karma: 15014853
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
No cant say I've seen it before. If the website is working with a bunch of forwards, i suggest you use the last URL in the chain to start the ogin process instead.
kovidgoyal is offline   Reply With Quote
Old 09-01-2019, 09:09 AM   #27
bobbysteel
Big Poppa
bobbysteel began at the beginning.
 
Posts: 103
Karma: 10
Join Date: Jul 2010
Device: Nook
looking at it, it seems to be using the same auth0 as WSJ, but the initial URL forward is just not forwarding as expected. there's no forwarding disabled, but there are a sequence of forwards that occur normally which are too complex for me to comprehend, but seemingly similar to teh WSJ. why wouldn't the forwrading work though? isn't beautifulsoup automatically forwarding?
bobbysteel is offline   Reply With Quote
Old 09-01-2019, 09:16 AM   #28
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,093
Karma: 15014853
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
the relevant software is mechanize, not beautifulsoup and in a browser forwarding can be done in any number of ways, princially using javascript in which case it would not work in mechanize, since it does not understand javascript.
kovidgoyal is offline   Reply With Quote
Old 09-01-2019, 09:17 AM   #29
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,093
Karma: 15014853
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
There JS heavy logins are a huge pain but once calibre 4 is released it should by easy to automate them using a headless Qt WebEngine instance. Do the login in the browser and extract the cookies and insert them into mechanize.
kovidgoyal is offline   Reply With Quote
Old 09-01-2019, 03:44 PM   #30
bobbysteel
Big Poppa
bobbysteel began at the beginning.
 
Posts: 103
Karma: 10
Join Date: Jul 2010
Device: Nook
can you point me to a recipe that does that? i just spent an hour trying to do this in python and pulling my hair out. i should have just done this all in nodejs i think myself mechanize is my kryptonite.
bobbysteel is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
New Yorker recipe failing adfadfsasdfafafd Recipes 2 03-28-2017 03:47 PM
"The New York Times" recipe failing with error mikebw Recipes 8 10-02-2015 06:48 PM
New York Times Recipe failing to verify SSL Cert Ramblurr Recipes 6 02-27-2015 05:31 AM
download of wsj failing amritsari Calibre 4 09-06-2012 06:53 PM
Economist Recipe Failing... awitko Recipes 2 11-07-2011 12:47 AM


All times are GMT -4. The time now is 08:36 PM.


MobileRead.com is a privately owned, operated and funded community.