Referrer problem with recipe

[Starson17]

A few months ago, I needed to send a referrer in a recipe. Kovid advised that Calibre sends no referrer, but I could monkey with mechanize to force it. Based on his code, I came up with this:

    def get_browser(self):
        br = BasicNewsRecipe.get_browser(self)
        orig_open_novisit = br.open_novisit
        def my_open_no_visit(url, **kwargs):
            req = mechanize.Request(
                    url,
                    headers = {
                        'Referer':'http://www.gocomics.com/',
                        })
            return orig_open_novisit(req)
        br.open_novisit = my_open_no_visit
        return br

(BTW, the header spells it "Referer") This code works fine more than 99.9% of the time, but on big recipes, that's not good enough. The whole recipe fails if it has an error, so eventually, it fails. I've ignored it, as I don't really need the large recipe to run very often - it's OK if it only runs every other time or so, and it does. The failures are inconsistent as far as I can tell. Here is the error:

Python function terminated unexpectedly
   (Error Code: 1)
Traceback (most recent call last):
  File "site.py", line 103, in main
  File "site.py", line 85, in run_entry_point
  File "C:\Util\Calibre2\src\src\calibre\utils\ipc\worker.py", line 99, in main
    result = func(*args, **kwargs)
  File "C:\Util\Calibre2\src\src\calibre\gui2\convert\gui_conversion.py", line 24, in gui_convert
    plumber.run()
  File "C:\Util\Calibre2\src\src\calibre\ebooks\conversion\plumber.py", line 815, in run
    accelerators, tdir)
  File "C:\Util\Calibre2\src\src\calibre\customize\conversion.py", line 211, in __call__
    log, accelerators)
  File "C:\Util\Calibre2\src\src\calibre\web\feeds\input.py", line 104, in convert
    ro.download()
  File "C:\Util\Calibre2\src\src\calibre\web\feeds\news.py", line 702, in download
    res = self.build_index()
  File "C:\Util\Calibre2\src\src\calibre\web\feeds\news.py", line 851, in build_index
    feeds = feeds_from_index(self.parse_index(), oldest_article=self.oldest_article,
  File "c:\users\appdata\local\temp\calibre_0.7.4_qzoqyx_recipes\recipe0.py", line 279, in parse_index
    articles = self.make_links(url)
  File "c:\users\appdata\local\temp\calibre_0.7.4_qzoqyx_recipes\recipe0.py", line 291, in make_links
    page_soup = self.index_to_soup(url)
  File "C:\Util\Calibre2\src\src\calibre\web\feeds\news.py", line 474, in index_to_soup
    with closing(open_func(url_or_raw)) as f:
  File "c:\users\appdata\local\temp\calibre_0.7.4_qzoqyx_recipes\recipe0.py", line 52, in my_open_no_visit
    return orig_open_novisit(req)
  File "site-packages\mechanize-0.1.11-py2.6.egg\mechanize\_mechanize.py", line 205, in open_novisit
  File "site-packages\mechanize-0.1.11-py2.6.egg\mechanize\_mechanize.py", line 261, in _mech_open
mechanize._response.httperror_seek_wrapper: HTTP Error 502: Proxy Error

Info on the HTTP Error 502: Proxy Error (bad gateway) is sparse. Does anyone have an idea how to solve this, or if not solve it, how to isolate it so that the entire recipe does not fail? By breaking the recipe down into smaller recipes, I don't see this problem as often.

[kovidgoyal]

Are you using a proxy? That error code is for proxy related errors

[Starson17]

Originally Posted by kovidgoyal

Are you using a proxy? That error code is for proxy related errors

No. I just doublechecked to make sure I hadn't inadvertently set a proxy in Windows, but no, there's nothing there. I was guessing that maybe somewhere in the mechanize operations, it looked like a proxy? This only occurs in recipes that use this referrer trick (I've got several recipes, but they all go to the same site).

I doubt it helps any, but I've been tracking this error more closely, and I've seen it several times, but I also caught this one for the same recipe:

  File "site-packages\mechanize-0.1.11-py2.6.egg\mechanize\_mechanize.py", line 261, in _mech_open
mechanize._response.httperror_seek_wrapper: HTTP Error 404: Not Found

[kovidgoyal]

I'm afraid I'm not exactly a mechanize guru, so I have no idea what would cause a 502 error other than a proxy.

A 404 error also (typically) indicates a URL pointing to something that does not exist.

[Starson17]

Any thoughts on error trapping to let the recipe complete?

[kovidgoyal]

One simple hack is to change my_open_novisit to try everything twice witha sleep of a second or two, using a try: except: block, in case the failure is temporary.

[Starson17]

Originally Posted by kovidgoyal

One simple hack...

That may do it. I've just run the big recipe twice in a row with no failures, but an hour ago - it failed. Perhaps simply retrying will fix it.

[Starson17]

Originally Posted by Starson17

Perhaps simply retrying will fix it.

That did it. Thanks for the suggestion.