Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Recipes

Notices

Reply
 
Thread Tools Search this Thread
Old 09-01-2019, 03:44 PM   #31
bobbysteel
Big Poppa
bobbysteel began at the beginning.
 
Posts: 103
Karma: 10
Join Date: Jul 2010
Device: Nook
this doesn't work https://stackoverflow.com/questions/...thon-mechanize
bobbysteel is offline   Reply With Quote
Old 09-01-2019, 11:34 PM   #32
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,099
Karma: 15014853
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Since calibre 4 is not released yet, there are no recipes that do that, but if you are asking how to set cookies on a mechanize broser simply call either set_cookie() with the string from the Set-Cookie header or alternately set_simple_cookie() with the name, domain and path of the cookie.
kovidgoyal is offline   Reply With Quote
Advert
Old 09-02-2019, 03:33 PM   #33
bobbysteel
Big Poppa
bobbysteel began at the beginning.
 
Posts: 103
Karma: 10
Join Date: Jul 2010
Device: Nook
would a mechanize cookie jar work? I'm trying this now and dumping my full cookies.txt but it's not authing.
bobbysteel is offline   Reply With Quote
Old 09-02-2019, 11:08 PM   #34
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,099
Karma: 15014853
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
If set_cookie is working then a cookie jar is not going to work either.
kovidgoyal is offline   Reply With Quote
Old 09-03-2019, 03:55 AM   #35
bobbysteel
Big Poppa
bobbysteel began at the beginning.
 
Posts: 103
Karma: 10
Join Date: Jul 2010
Device: Nook
Quote:
Originally Posted by kovidgoyal View Post
If set_cookie is working then a cookie jar is not going to work either.
i can't try set cookie as there are over a dozen fields and i don't have the hours to rewrite each into set cookie format. the cookies.txt should work but i don't know how to even do a log of it in Python to fix this. in JS I'd juse JSON.stringify but when ido a self.log on the cookie jar it just shows object of type MozillaCookieJar but can't even tell the contents are being loaded
bobbysteel is offline   Reply With Quote
Advert
Old 09-03-2019, 04:07 AM   #36
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,099
Karma: 15014853
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
If the cookies are being loaded, then they would be present as Cookie headers when you use set_debug_http(). If you want to check if a jar has cookies, simply use save() to save it to a file and inspect that.
kovidgoyal is offline   Reply With Quote
Old 09-03-2019, 11:07 AM   #37
bobbysteel
Big Poppa
bobbysteel began at the beginning.
 
Posts: 103
Karma: 10
Join Date: Jul 2010
Device: Nook
I can see SOME of the cookies in the debug but they're not all getting passed through

Code:
       kw['user_agent'] = random_user_agent(allow_ie=False)
        br = BasicNewsRecipe.get_browser(self, *a, **kw)
        self.log('Starting login process...')
        br.set_debug_redirects(True)
        br.set_debug_http(True)
        cookiejar = mechanize.FileCookieJar('/home/me/calibre-commandline/cookies.txt')
        #br.set_handle_robots(False)
        br.set_cookiejar(cookiejar)                                                                                             #self.log(cookiejar)
        res = br.open(start_url)
this fails to login. but just curl'ing with curl -c cookies.txt <paywalled url> works fine.

looking at the debug headers some of the cookie headers are being passed but not all of them. the cookie file looks fine though.
bobbysteel is offline   Reply With Quote
Old 09-03-2019, 12:38 PM   #38
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,099
Karma: 15014853
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Whether a cookie gets picked up or not depends on its path, expiration date, domain, etc. It should be fairly easy to figure out which is th eimportant cookie by doing a binary search on cookies.txt with curl and then manually setting that one using set_simple_cookie() which should avoid issues with expiration dates, etc.
kovidgoyal is offline   Reply With Quote
Old 09-04-2019, 11:31 AM   #39
bobbysteel
Big Poppa
bobbysteel began at the beginning.
 
Posts: 103
Karma: 10
Join Date: Jul 2010
Device: Nook
Hm. So with curl I've isolated it to two cookies acs_tnl and sacs_tnl
Manually calling with curl works fine reading the cookies.txt with just those.
However in the recipe
Code:
        br.set_debug_redirects(True)
        br.set_debug_http(True)
        br.addheaders = [('Cookie','acs_tnl=xxxxxxxxx')]       
        br.addheaders = [('Cookie','sacs_tnl=f')]
still fails.
Curl with just those two cookies also seems to fail. I can't seem to figure out why the same text file works as a file but not manually.

Last edited by bobbysteel; 09-04-2019 at 11:36 AM.
bobbysteel is offline   Reply With Quote
Old 09-04-2019, 11:45 AM   #40
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,099
Karma: 15014853
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Hmm well if it is not working with curl with those two cookies, it's not likely to work with mechanize either.
kovidgoyal is offline   Reply With Quote
Old 09-04-2019, 01:18 PM   #41
bobbysteel
Big Poppa
bobbysteel began at the beginning.
 
Posts: 103
Karma: 10
Join Date: Jul 2010
Device: Nook
ah i've realized there's also some antisharing mechanism now kicking in
bobbysteel is offline   Reply With Quote
Old 09-17-2019, 10:58 AM   #42
bobbysteel
Big Poppa
bobbysteel began at the beginning.
 
Posts: 103
Karma: 10
Join Date: Jul 2010
Device: Nook
Quote:
Originally Posted by kovidgoyal View Post
Hmm well if it is not working with curl with those two cookies, it's not likely to work with mechanize either.
it's weird - the auth0 system is rather complex. I've tried using headless chrome to generate a cookie file and then using that in the recipe, but won't work. for your v4 recipes, how will they differ in cases like this? i'm running your 3.99 beta and would like to play around with it.
cheers
bobbysteel is offline   Reply With Quote
Old 09-17-2019, 11:07 AM   #43
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,099
Karma: 15014853
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
The recipes will be no different. You simply have the option of running a headless browser to extract cookies. And eventually I may add some convenience APIs to download the entire content using a headless browser. No need to use mechanize at all.
kovidgoyal is offline   Reply With Quote
Old 09-17-2019, 11:38 AM   #44
bobbysteel
Big Poppa
bobbysteel began at the beginning.
 
Posts: 103
Karma: 10
Join Date: Jul 2010
Device: Nook
Quote:
Originally Posted by kovidgoyal View Post
The recipes will be no different. You simply have the option of running a headless browser to extract cookies. And eventually I may add some convenience APIs to download the entire content using a headless browser. No need to use mechanize at all.
truly weird. with headless chrome i can automate the login no problem, but mechanize doesn't seem to work even with the same cookies set post-login. any ideas what else would differ between the two tools?
bobbysteel is offline   Reply With Quote
Old 09-17-2019, 12:12 PM   #45
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 36,099
Karma: 15014853
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
javascript can set arbitrary headers, not just cookies. So you have to resort to exaing the actual HTTP requests to see the difference.
kovidgoyal is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
New Yorker recipe failing adfadfsasdfafafd Recipes 2 03-28-2017 03:47 PM
"The New York Times" recipe failing with error mikebw Recipes 8 10-02-2015 06:48 PM
New York Times Recipe failing to verify SSL Cert Ramblurr Recipes 6 02-27-2015 05:31 AM
download of wsj failing amritsari Calibre 4 09-06-2012 06:53 PM
Economist Recipe Failing... awitko Recipes 2 11-07-2011 12:47 AM


All times are GMT -4. The time now is 05:45 AM.


MobileRead.com is a privately owned, operated and funded community.