debug mechanize

[Kleki]

Hi,

I tried to add a login to the FAZ recipe. The added code seems to work in principle. I had some error messages related to syntax I was able to fix. When I change the form ID to something other than the actual ID I get an error message, so I believe the form is processed.

However, the downloaded news are the ones for not logged in users (only showing the first couple of paragraphs).

Is there a way to log what is sent forth and back between Calibre and the news site, the http requests and responses? Can I view the saved cookies somewhere?

Appreciate any hint on how to debug my recipe.

[kovidgoyal]

https://mechanize.readthedocs.io/en/...l#http-headers

[Kleki]

Thanks Kovid,

I'm novice to Calibre. While I have some history of programming this is my first exposure to Python and I don't have any development environment set up here.

I'm just using the UI to run the recipe and see the result. Adding those lines I believe you suggested does not add any additional data to the output in the jobs overview.

I simply added this to the recipe, not sure if I should have done any modification:

import sys, logging
logger = logging.getLogger("mechanize")
logger.addHandler(logging.StreamHandler(sys.stdout))
logger.setLevel(logging.DEBUG)

I'd appreciate any help in how and where I see the debug output or some instructions if I need to use any command line or debug mode to be able to do so.

[kovidgoyal]

You need to override get_browser() in the recipe, like this:

    def get_browser(self, *args, **kwargs):
        browser = BasicNewsrecipe.get_browser(self, *args, **kwargs)
        browser.set_debug_http(True)
        browser.set_debug_responses(True)
        browser.set_debug_redirects(True)
        return browser