06-26-2010, 09:32 AM | #1 |
Wizard
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
|
Referrer problem with recipe
A few months ago, I needed to send a referrer in a recipe. Kovid advised that Calibre sends no referrer, but I could monkey with mechanize to force it. Based on his code, I came up with this:
Code:
def get_browser(self): br = BasicNewsRecipe.get_browser(self) orig_open_novisit = br.open_novisit def my_open_no_visit(url, **kwargs): req = mechanize.Request( url, headers = { 'Referer':'http://www.gocomics.com/', }) return orig_open_novisit(req) br.open_novisit = my_open_no_visit return br Code:
Python function terminated unexpectedly (Error Code: 1) Traceback (most recent call last): File "site.py", line 103, in main File "site.py", line 85, in run_entry_point File "C:\Util\Calibre2\src\src\calibre\utils\ipc\worker.py", line 99, in main result = func(*args, **kwargs) File "C:\Util\Calibre2\src\src\calibre\gui2\convert\gui_conversion.py", line 24, in gui_convert plumber.run() File "C:\Util\Calibre2\src\src\calibre\ebooks\conversion\plumber.py", line 815, in run accelerators, tdir) File "C:\Util\Calibre2\src\src\calibre\customize\conversion.py", line 211, in __call__ log, accelerators) File "C:\Util\Calibre2\src\src\calibre\web\feeds\input.py", line 104, in convert ro.download() File "C:\Util\Calibre2\src\src\calibre\web\feeds\news.py", line 702, in download res = self.build_index() File "C:\Util\Calibre2\src\src\calibre\web\feeds\news.py", line 851, in build_index feeds = feeds_from_index(self.parse_index(), oldest_article=self.oldest_article, File "c:\users\appdata\local\temp\calibre_0.7.4_qzoqyx_recipes\recipe0.py", line 279, in parse_index articles = self.make_links(url) File "c:\users\appdata\local\temp\calibre_0.7.4_qzoqyx_recipes\recipe0.py", line 291, in make_links page_soup = self.index_to_soup(url) File "C:\Util\Calibre2\src\src\calibre\web\feeds\news.py", line 474, in index_to_soup with closing(open_func(url_or_raw)) as f: File "c:\users\appdata\local\temp\calibre_0.7.4_qzoqyx_recipes\recipe0.py", line 52, in my_open_no_visit return orig_open_novisit(req) File "site-packages\mechanize-0.1.11-py2.6.egg\mechanize\_mechanize.py", line 205, in open_novisit File "site-packages\mechanize-0.1.11-py2.6.egg\mechanize\_mechanize.py", line 261, in _mech_open mechanize._response.httperror_seek_wrapper: HTTP Error 502: Proxy Error |
06-26-2010, 10:59 AM | #2 |
creator of calibre
Posts: 43,778
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Are you using a proxy? That error code is for proxy related errors
|
Advert | |
|
06-26-2010, 11:54 AM | #3 | |
Wizard
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
|
Quote:
I doubt it helps any, but I've been tracking this error more closely, and I've seen it several times, but I also caught this one for the same recipe: Code:
File "site-packages\mechanize-0.1.11-py2.6.egg\mechanize\_mechanize.py", line 261, in _mech_open mechanize._response.httperror_seek_wrapper: HTTP Error 404: Not Found |
|
06-26-2010, 12:00 PM | #4 |
creator of calibre
Posts: 43,778
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
I'm afraid I'm not exactly a mechanize guru, so I have no idea what would cause a 502 error other than a proxy.
A 404 error also (typically) indicates a URL pointing to something that does not exist. |
06-26-2010, 12:09 PM | #5 |
Wizard
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
|
Any thoughts on error trapping to let the recipe complete?
|
Advert | |
|
06-26-2010, 12:23 PM | #6 |
creator of calibre
Posts: 43,778
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
One simple hack is to change my_open_novisit to try everything twice witha sleep of a second or two, using a try: except: block, in case the failure is temporary.
|
06-26-2010, 12:28 PM | #7 |
Wizard
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
|
|
06-26-2010, 04:47 PM | #8 |
Wizard
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
|
|
Thread Tools | Search this Thread |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
New recipe | kiklop74 | Recipes | 0 | 10-01-2010 02:42 PM |
Catholic News Recipe Problem | funkgut | Calibre | 4 | 04-23-2010 02:08 PM |
Recipe problem on Kindle DX | jbambridge | Calibre | 9 | 08-06-2009 01:24 PM |
Recipe Help Please | estral | Calibre | 1 | 06-11-2009 02:35 PM |
A problem with custom recipe | joshdu1125 | Calibre | 1 | 05-03-2009 09:31 PM |