Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 06-26-2010, 09:32 AM   #1
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Referrer problem with recipe

A few months ago, I needed to send a referrer in a recipe. Kovid advised that Calibre sends no referrer, but I could monkey with mechanize to force it. Based on his code, I came up with this:
Code:
    def get_browser(self):
        br = BasicNewsRecipe.get_browser(self)
        orig_open_novisit = br.open_novisit
        def my_open_no_visit(url, **kwargs):
            req = mechanize.Request(
                    url,
                    headers = {
                        'Referer':'http://www.gocomics.com/',
                        })
            return orig_open_novisit(req)
        br.open_novisit = my_open_no_visit
        return br
(BTW, the header spells it "Referer") This code works fine more than 99.9% of the time, but on big recipes, that's not good enough. The whole recipe fails if it has an error, so eventually, it fails. I've ignored it, as I don't really need the large recipe to run very often - it's OK if it only runs every other time or so, and it does. The failures are inconsistent as far as I can tell. Here is the error:
Code:
Python function terminated unexpectedly
   (Error Code: 1)
Traceback (most recent call last):
  File "site.py", line 103, in main
  File "site.py", line 85, in run_entry_point
  File "C:\Util\Calibre2\src\src\calibre\utils\ipc\worker.py", line 99, in main
    result = func(*args, **kwargs)
  File "C:\Util\Calibre2\src\src\calibre\gui2\convert\gui_conversion.py", line 24, in gui_convert
    plumber.run()
  File "C:\Util\Calibre2\src\src\calibre\ebooks\conversion\plumber.py", line 815, in run
    accelerators, tdir)
  File "C:\Util\Calibre2\src\src\calibre\customize\conversion.py", line 211, in __call__
    log, accelerators)
  File "C:\Util\Calibre2\src\src\calibre\web\feeds\input.py", line 104, in convert
    ro.download()
  File "C:\Util\Calibre2\src\src\calibre\web\feeds\news.py", line 702, in download
    res = self.build_index()
  File "C:\Util\Calibre2\src\src\calibre\web\feeds\news.py", line 851, in build_index
    feeds = feeds_from_index(self.parse_index(), oldest_article=self.oldest_article,
  File "c:\users\appdata\local\temp\calibre_0.7.4_qzoqyx_recipes\recipe0.py", line 279, in parse_index
    articles = self.make_links(url)
  File "c:\users\appdata\local\temp\calibre_0.7.4_qzoqyx_recipes\recipe0.py", line 291, in make_links
    page_soup = self.index_to_soup(url)
  File "C:\Util\Calibre2\src\src\calibre\web\feeds\news.py", line 474, in index_to_soup
    with closing(open_func(url_or_raw)) as f:
  File "c:\users\appdata\local\temp\calibre_0.7.4_qzoqyx_recipes\recipe0.py", line 52, in my_open_no_visit
    return orig_open_novisit(req)
  File "site-packages\mechanize-0.1.11-py2.6.egg\mechanize\_mechanize.py", line 205, in open_novisit
  File "site-packages\mechanize-0.1.11-py2.6.egg\mechanize\_mechanize.py", line 261, in _mech_open
mechanize._response.httperror_seek_wrapper: HTTP Error 502: Proxy Error
Info on the HTTP Error 502: Proxy Error (bad gateway) is sparse. Does anyone have an idea how to solve this, or if not solve it, how to isolate it so that the entire recipe does not fail? By breaking the recipe down into smaller recipes, I don't see this problem as often.
Starson17 is offline   Reply With Quote
Old 06-26-2010, 10:59 AM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,778
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Are you using a proxy? That error code is for proxy related errors
kovidgoyal is offline   Reply With Quote
Advert
Old 06-26-2010, 11:54 AM   #3
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by kovidgoyal View Post
Are you using a proxy? That error code is for proxy related errors
No. I just doublechecked to make sure I hadn't inadvertently set a proxy in Windows, but no, there's nothing there. I was guessing that maybe somewhere in the mechanize operations, it looked like a proxy? This only occurs in recipes that use this referrer trick (I've got several recipes, but they all go to the same site).

I doubt it helps any, but I've been tracking this error more closely, and I've seen it several times, but I also caught this one for the same recipe:
Code:
  File "site-packages\mechanize-0.1.11-py2.6.egg\mechanize\_mechanize.py", line 261, in _mech_open
mechanize._response.httperror_seek_wrapper: HTTP Error 404: Not Found
Starson17 is offline   Reply With Quote
Old 06-26-2010, 12:00 PM   #4
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,778
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
I'm afraid I'm not exactly a mechanize guru, so I have no idea what would cause a 502 error other than a proxy.

A 404 error also (typically) indicates a URL pointing to something that does not exist.
kovidgoyal is offline   Reply With Quote
Old 06-26-2010, 12:09 PM   #5
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Any thoughts on error trapping to let the recipe complete?
Starson17 is offline   Reply With Quote
Advert
Old 06-26-2010, 12:23 PM   #6
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,778
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
One simple hack is to change my_open_novisit to try everything twice witha sleep of a second or two, using a try: except: block, in case the failure is temporary.
kovidgoyal is offline   Reply With Quote
Old 06-26-2010, 12:28 PM   #7
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by kovidgoyal View Post
One simple hack...
That may do it. I've just run the big recipe twice in a row with no failures, but an hour ago - it failed. Perhaps simply retrying will fix it.
Starson17 is offline   Reply With Quote
Old 06-26-2010, 04:47 PM   #8
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by Starson17 View Post
Perhaps simply retrying will fix it.
That did it. Thanks for the suggestion.
Starson17 is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
New recipe kiklop74 Recipes 0 10-01-2010 02:42 PM
Catholic News Recipe Problem funkgut Calibre 4 04-23-2010 02:08 PM
Recipe problem on Kindle DX jbambridge Calibre 9 08-06-2009 01:24 PM
Recipe Help Please estral Calibre 1 06-11-2009 02:35 PM
A problem with custom recipe joshdu1125 Calibre 1 05-03-2009 09:31 PM


All times are GMT -4. The time now is 07:24 AM.


MobileRead.com is a privately owned, operated and funded community.