[Starson17]

Originally Posted by krunk

I'm getting the following error in a custom recipe:
HTTP Error 403: Bad Behavior

Spoiler:

1% Fetching feed Skeptic Blog...
Failed feed: Skeptic Blog
Traceback (most recent call last):
  File "site-packages/calibre/web/feeds/news.py", line 1274, in parse_feeds
  File "site-packages/mechanize/_mechanize.py", line 209, in open
  File "site-packages/mechanize/_mechanize.py", line 261, in _mech_open
httperror_seek_wrapper: HTTP Error 403: Bad Behavior

I overloaded the get_browser() method with my own.

def get_browser(self):
    br = BasicNewsRecipe.get_browser()
    br.open(my_url)
    return br

From this I was able to determine it was throwing the 403 Bad Behavior error on mechanize.Browser().open().

This doesn't occur with a vanilla mechanize browser. Right now I'm doing:

def get_browser(self):
    br = mechanize.Browser()
    br.open(my_url)
    return br

This throws no errors and seems to parse well. My question is what in the BasicNewsRecipe's browser could be causing the above error? Is there a more correct or better way of resolving the problem?

It could be several things, depending on how the Bad Behavior module is set. I was able to narrow it down to a single header and pass the test for one site by sending a simple Accept: header to avoid being seen as a spambot. You can try what I did:
https://www.mobileread.com/forums/sho...postcount=2399
Or you can use TamperData to track down exactly why FireFox passes the test and Calibre does not.