An existing connection was forcibly closed

[RobFreundlich]

This morning, a custom recipe I've been using for a few months failed with "An existing connection was forcibly closed by the remote host". The recipe simulates a login to the site's daily newspaper and then scrapes the HTML, so I tried manually logging in, and everything works fine.

Does anyone have any ideas what I can do to debug this? Is there a way to get more details about the error or to tell why the force-close happened?

I've attached the recipe (it requires a username and password to the Boston Globe). Here's the failure trace:

Python function terminated unexpectedly
  <urlopen error [Errno 10054] An existing connection was forcibly closed by the remote host> (Error Code: 1)
Traceback (most recent call last):
  File "site.py", line 132, in main
  File "site.py", line 109, in run_entry_point
  File "site-packages\calibre\ebooks\conversion\cli.py", line 325, in main
  File "site-packages\calibre\ebooks\conversion\plumber.py", line 979, in run
  File "site-packages\calibre\customize\conversion.py", line 208, in __call__
  File "site-packages\calibre\ebooks\conversion\plugins\recipe_input.py", line 104, in convert
  File "site-packages\calibre\web\feeds\news.py", line 770, in __init__
  File "<string>", line 61, in get_browser
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_mechanize.py", line 541, in submit
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_mechanize.py", line 203, in open
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_mechanize.py", line 230, in _mech_open
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_opener.py", line 204, in open
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_urllib2_fork.py", line 457, in http_response
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_opener.py", line 221, in error
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_urllib2_fork.py", line 332, in _call_chain
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_urllib2_fork.py", line 571, in http_error_302
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_mechanize.py", line 203, in open
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_mechanize.py", line 230, in _mech_open
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_opener.py", line 204, in open
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_urllib2_fork.py", line 457, in http_response
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_opener.py", line 221, in error
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_urllib2_fork.py", line 332, in _call_chain
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_urllib2_fork.py", line 571, in http_error_302
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_mechanize.py", line 203, in open
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_mechanize.py", line 230, in _mech_open
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_opener.py", line 193, in open
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_urllib2_fork.py", line 344, in _open
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_urllib2_fork.py", line 332, in _call_chain
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_urllib2_fork.py", line 1142, in http_open
  File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_urllib2_fork.py", line 1118, in do_open
urllib2.URLError: <urlopen error [Errno 10054] An existing connection was forcibly closed by the remote host>

[RobFreundlich]

Here's a couple more pieces of the puzzle:

1. The login page has mixed secure and non-secure content - if you browse to it you get a warning message (popup dialog in IE, yellow bar in Chrome).

2. If I run the recipe while Fiddler (a web debugger/proxy) is running, the recipe runs successfully.

It's almost as if the mixed content is confusing the BeautifulSoup browser and somehow Fiddler is ameliorating that confusion. But I don't have any idea how to go any further (I suppose I could leave Fiddler running every night for now, but blech!)

[RobFreundlich]

The solution to this problem turned out to be "call the Boston Globe".

I ran Fiddler last night so my download could get past the error, and when I looked at the paper this morning, most of the articles read "You have exceeded your monthly allocation of free articles. Please create an account or log in." Since I have an account, something was clearly amiss.

I called the Globe, and they said that Wednesday night they had done a "system wide purge, so any account with a glitch is showing up now". Speaking as someone who's done tech support, I can honestly say that sounds like code for "we screwed something up and are going to try to dazzle the customer with technobabble while we fix their problem." They fixed my logon, and everything works again.

So, to any future reader who is finding this thread in the hope of getting a solution to a remotely closed connection, I apologize, because there's no technical solution here. But maybe your solution will be as simple as mine.