mechanize.py error when logging in to subscribed articles

janvanmaar · 11-02-2010, 09:34 AM

I try to use the following plugin to fetch subscribed articles:

Code:

from calibre.web.feeds.news import BasicNewsRecipe

class TyzdenRecipe(BasicNewsRecipe):
    title          = u'.tyzden'
    oldest_article = 7
    max_articles_per_feed = 100
    needs_subscription    = True
    language = 'sk'

    feeds          = [(u'.tyzden - clanky z tlacenych vydani', u'http://www.tyzden.sk/rss/clanky-z-tlacenych-vydani.xml')]

    LOGIN = 'http://www.tyzden.sk/prihlasenie.html'

    def get_browser(self):
        br = BasicNewsRecipe.get_browser()
        if self.username is not None and self.password is not None:
            br.open(self.LOGIN)
            br.select_form(nr=1)
            br['user'] = self.username
            br['pass'] = self.password
            br.submit()
        return br

It gets to the right form and finds the right fields but then crashes when trying to submit the form:

Quote:

InputFormatPlugin: Recipe Input running
Traceback (most recent call last):
File "/tmp/init.py", line 48, in <module>
File "/home/kovid/build/calibre/src/calibre/utils/ipc/worker.py", line 107, in main
File "/home/kovid/build/calibre/src/calibre/gui2/convert/gui_conversion.py", line 24, in gui_convert
File "/home/kovid/build/calibre/src/calibre/ebooks/conversion/plumber.py", line 836, in run
File "/home/kovid/build/calibre/src/calibre/customize/conversion.py", line 216, in __call__
File "/home/kovid/build/calibre/src/calibre/web/feeds/input.py", line 101, in convert
File "/home/kovid/build/calibre/src/calibre/web/feeds/news.py", line 616, in __init__
File "/tmp/calibre_0.7.26_tmp_bsIqcm/calibre_0.7.26_u79lhp_recipes/recipe0.py", line 29, in get_browser
br.submit()
File "/usr/lib64/python2.6/site-packages/mechanize/_mechanize.py", line 547, in submit
File "/usr/lib64/python2.6/site-packages/mechanize/_mechanize.py", line 209, in open
File "/usr/lib64/python2.6/site-packages/mechanize/_mechanize.py", line 261, in _mech_open
mechanize._response.httperror_seek_wrapper: HTTP Error 302: The HTTP server returned a redirect error that would lead to an infinite loop.
The last 30x error message was:
Found

I don't understand why this is happening... Any idea?
I should mention that login works fine when I fill the form in a browser. I double checked that correct username and password are inputted in the scheduler also...

kovidgoyal · 11-02-2010, 11:47 AM

that means mechanize thinks there is an infinite redirect loop, which typically means you aren't filling in the form correctly. There may be hidden fields that need to be filled in. Use Tamper data in firefox to see what they might be.

janvanmaar · 11-02-2010, 12:51 PM

Thanks for the Tamper suggestion!
It did not reveal any required hidden fields. But it showed that after the form has been submitted, the browser asked for logout and then for login again with a cookie... and when I disabled cookies then login did not work.
I guess this is very difficult to do in Calibre/python, isn't it... but at least I know where the problem is.

kovidgoyal · 11-02-2010, 01:02 PM

mechanize handles cookies just fine. You should set handle refresh to false on the mechanize browser and simulate the logout/login steps manually with br.open

janvanmaar · 11-02-2010, 01:33 PM

Hm, I guess I should have said it is difficult for me as somebody without much knowledge of HTTP protocol.
Anyway, whatever I do, I get the same problem when submitting the form. Setting handle refresh to false does not change that. And I cannot do other br.open as it crashes before...

kovidgoyal · 11-02-2010, 01:41 PM

use a try catch to catch the exception and ignore it.

janvanmaar · 11-02-2010, 02:04 PM

well, that enabled me to pass through the submit (which does not mean that registration was successful of course), however, I got a segfault when creating the ebook then:
calibre-parallel: line 11: 17597 Segmentation fault $loader "$@"
EDIT: I am using binaries - perhaps I should try to compile...

janvanmaar · 11-02-2010, 03:45 PM

Guess what: it worked! I would not say that ignoring the error could be working but it is. Thanks for all your help!

kovidgoyal · 11-02-2010, 03:46 PM

the segfault can be ignored, it happens during interpreter shutdown and is an artifact of the process of creating the binaries that I have never been motivated enough to track down and fix.

janvanmaar · 11-02-2010, 04:10 PM

Takehome message from this thread: ignore errors and conquer.

11-02-2010, 02:04 PM	#7
janvanmaar Addict Posts: 219 Karma: 404 Join Date: Nov 2010 Device: Kindle 3G, Samsung SIII	well, that enabled me to pass through the submit (which does not mean that registration was successful of course), however, I got a segfault when creating the ebook then: calibre-parallel: line 11: 17597 Segmentation fault $loader "$@" EDIT: I am using binaries - perhaps I should try to compile... Last edited by janvanmaar; 11-02-2010 at 02:08 PM.

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Search within subscribed threads!	thinkpad	Feedback	3	08-18-2010 09:44 AM
I don't get email notifications of updates to threads I'm subscribed to	Ea	Feedback	4	03-26-2010 02:20 PM
calibre-0.6.31, mechanize and lxml	taurnil	Calibre	5	01-01-2010 07:47 AM
Views subscribed threads	pilotbob	Feedback	6	02-28-2009 02:49 PM
'mechanize' module?	itimpi	Calibre	2	01-22-2009 03:42 PM

11-02-2010, 11:47 AM	#2
kovidgoyal creator of calibre Posts: 45,209 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	that means mechanize thinks there is an infinite redirect loop, which typically means you aren't filling in the form correctly. There may be hidden fields that need to be filled in. Use Tamper data in firefox to see what they might be.

11-02-2010, 12:51 PM	#3
janvanmaar Addict Posts: 219 Karma: 404 Join Date: Nov 2010 Device: Kindle 3G, Samsung SIII	Thanks for the Tamper suggestion! It did not reveal any required hidden fields. But it showed that after the form has been submitted, the browser asked for logout and then for login again with a cookie... and when I disabled cookies then login did not work. I guess this is very difficult to do in Calibre/python, isn't it... but at least I know where the problem is.

11-02-2010, 01:02 PM	#4
kovidgoyal creator of calibre Posts: 45,209 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	mechanize handles cookies just fine. You should set handle refresh to false on the mechanize browser and simulate the logout/login steps manually with br.open

11-02-2010, 01:33 PM	#5
janvanmaar Addict Posts: 219 Karma: 404 Join Date: Nov 2010 Device: Kindle 3G, Samsung SIII	Hm, I guess I should have said it is difficult for me as somebody without much knowledge of HTTP protocol. Anyway, whatever I do, I get the same problem when submitting the form. Setting handle refresh to false does not change that. And I cannot do other br.open as it crashes before...

11-02-2010, 01:41 PM	#6
kovidgoyal creator of calibre Posts: 45,209 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	use a try catch to catch the exception and ignore it.

11-02-2010, 03:45 PM	#8
janvanmaar Addict Posts: 219 Karma: 404 Join Date: Nov 2010 Device: Kindle 3G, Samsung SIII	Guess what: it worked! I would not say that ignoring the error could be working but it is. Thanks for all your help!

11-02-2010, 03:46 PM	#9
kovidgoyal creator of calibre Posts: 45,209 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	the segfault can be ignored, it happens during interpreter shutdown and is an artifact of the process of creating the binaries that I have never been motivated enough to track down and fix.

11-02-2010, 04:10 PM	#10
janvanmaar Addict Posts: 219 Karma: 404 Join Date: Nov 2010 Device: Kindle 3G, Samsung SIII	Takehome message from this thread: ignore errors and conquer.

Advert

Advert