09-01-2019, 02:44 PM | #31 |
Big Poppa
Posts: 110
Karma: 10
Join Date: Jul 2010
Device: Nook
|
this doesn't work https://stackoverflow.com/questions/...thon-mechanize
|
09-01-2019, 10:34 PM | #32 |
creator of calibre
Posts: 43,844
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Since calibre 4 is not released yet, there are no recipes that do that, but if you are asking how to set cookies on a mechanize broser simply call either set_cookie() with the string from the Set-Cookie header or alternately set_simple_cookie() with the name, domain and path of the cookie.
|
Advert | |
|
09-02-2019, 02:33 PM | #33 |
Big Poppa
Posts: 110
Karma: 10
Join Date: Jul 2010
Device: Nook
|
would a mechanize cookie jar work? I'm trying this now and dumping my full cookies.txt but it's not authing.
|
09-02-2019, 10:08 PM | #34 |
creator of calibre
Posts: 43,844
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
If set_cookie is working then a cookie jar is not going to work either.
|
09-03-2019, 02:55 AM | #35 |
Big Poppa
Posts: 110
Karma: 10
Join Date: Jul 2010
Device: Nook
|
i can't try set cookie as there are over a dozen fields and i don't have the hours to rewrite each into set cookie format. the cookies.txt should work but i don't know how to even do a log of it in Python to fix this. in JS I'd juse JSON.stringify but when ido a self.log on the cookie jar it just shows object of type MozillaCookieJar but can't even tell the contents are being loaded
|
Advert | |
|
09-03-2019, 03:07 AM | #36 |
creator of calibre
Posts: 43,844
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
If the cookies are being loaded, then they would be present as Cookie headers when you use set_debug_http(). If you want to check if a jar has cookies, simply use save() to save it to a file and inspect that.
|
09-03-2019, 10:07 AM | #37 |
Big Poppa
Posts: 110
Karma: 10
Join Date: Jul 2010
Device: Nook
|
I can see SOME of the cookies in the debug but they're not all getting passed through
Code:
kw['user_agent'] = random_user_agent(allow_ie=False) br = BasicNewsRecipe.get_browser(self, *a, **kw) self.log('Starting login process...') br.set_debug_redirects(True) br.set_debug_http(True) cookiejar = mechanize.FileCookieJar('/home/me/calibre-commandline/cookies.txt') #br.set_handle_robots(False) br.set_cookiejar(cookiejar) #self.log(cookiejar) res = br.open(start_url) looking at the debug headers some of the cookie headers are being passed but not all of them. the cookie file looks fine though. |
09-03-2019, 11:38 AM | #38 |
creator of calibre
Posts: 43,844
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Whether a cookie gets picked up or not depends on its path, expiration date, domain, etc. It should be fairly easy to figure out which is th eimportant cookie by doing a binary search on cookies.txt with curl and then manually setting that one using set_simple_cookie() which should avoid issues with expiration dates, etc.
|
09-04-2019, 10:31 AM | #39 |
Big Poppa
Posts: 110
Karma: 10
Join Date: Jul 2010
Device: Nook
|
Hm. So with curl I've isolated it to two cookies acs_tnl and sacs_tnl
Manually calling with curl works fine reading the cookies.txt with just those. However in the recipe Code:
br.set_debug_redirects(True) br.set_debug_http(True) br.addheaders = [('Cookie','acs_tnl=xxxxxxxxx')] br.addheaders = [('Cookie','sacs_tnl=f')] Curl with just those two cookies also seems to fail. I can't seem to figure out why the same text file works as a file but not manually. Last edited by bobbysteel; 09-04-2019 at 10:36 AM. |
09-04-2019, 10:45 AM | #40 |
creator of calibre
Posts: 43,844
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Hmm well if it is not working with curl with those two cookies, it's not likely to work with mechanize either.
|
09-04-2019, 12:18 PM | #41 |
Big Poppa
Posts: 110
Karma: 10
Join Date: Jul 2010
Device: Nook
|
ah i've realized there's also some antisharing mechanism now kicking in
|
09-17-2019, 09:58 AM | #42 | |
Big Poppa
Posts: 110
Karma: 10
Join Date: Jul 2010
Device: Nook
|
Quote:
cheers |
|
09-17-2019, 10:07 AM | #43 |
creator of calibre
Posts: 43,844
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
The recipes will be no different. You simply have the option of running a headless browser to extract cookies. And eventually I may add some convenience APIs to download the entire content using a headless browser. No need to use mechanize at all.
|
09-17-2019, 10:38 AM | #44 |
Big Poppa
Posts: 110
Karma: 10
Join Date: Jul 2010
Device: Nook
|
truly weird. with headless chrome i can automate the login no problem, but mechanize doesn't seem to work even with the same cookies set post-login. any ideas what else would differ between the two tools?
|
09-17-2019, 11:12 AM | #45 |
creator of calibre
Posts: 43,844
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
javascript can set arbitrary headers, not just cookies. So you have to resort to exaing the actual HTTP requests to see the difference.
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
New Yorker recipe failing | adfadfsasdfafafd | Recipes | 2 | 03-28-2017 02:47 PM |
"The New York Times" recipe failing with error | mikebw | Recipes | 8 | 10-02-2015 05:48 PM |
New York Times Recipe failing to verify SSL Cert | Ramblurr | Recipes | 6 | 02-27-2015 04:31 AM |
download of wsj failing | amritsari | Calibre | 4 | 09-06-2012 05:53 PM |
Economist Recipe Failing... | awitko | Recipes | 2 | 11-06-2011 11:47 PM |