Plugin error: IOError: [Errno 32] Broken pipe

[stoduk]

Hi Folks,

I'm doing some development for a plugin (it is the X-Ray plugin that Matthew Wilson originally wrote), and I'm hitting a rather puzzling problem.

Copious notes below, but basically writing to see if:
- anyone is seeing similar issues with this or other plugins
- anyone has any ideas where to look for the problem

The symptom of the problem is simply the IOError(32, "Broken pipe") exception being raised when the plugin tries to write to stdout (directly or via print).

I have found various mentions of this exception relating to Calibre, but a lot of those seem to be caused by virus software causing problems getting webpages. I found one mention of a different case, but no solution or further information to go on.

Cheers,
Anthony

Sometimes I cannot reproduce the problem if I try (mostly when I've set aside some time to look at it!), the rest of the time I hit it very readily. The problem is never hit on the first run of the plugin, but is very often hit on the second and subsequent runs. It takes a restart of Calibre to get things working again.

The error is seen when a call to print or sys.stdout.write is made in that plugin - and depending on which of those is called I either get the IOError(32, "Broken pipe") mentioned in the subject or sometimes get an odd IOError(0) exception.

I've had a quick dig around the code to see where pipes are used - I'm assuming the problem is that whatever our plugin's stdout is connected to is has closed the pipe, hence the exception. However, "pipe" seems to be found a lot in the code - so really it seems more sensible to try to understand how the code works/where likely pipes are, rather than finding the bug via code inspection

[chaley]

If you are using windows as your development environment then the problem is probably caused by windows not being able to buffer and write output to the screen as fast as the program can generate it. Simply redirect standard output and standard error to a file and the problem goes away. For example, when debugging calibre using print statements that generate lots of output I must use:

calibre-debug -g > foo.txt 2>&1

[stoduk]

OK, should have given more details Calibre 2.44.1 running on OS X 10.10.5.

However, your suggestion might still hold on OS X too - the plugin has borrowed the kindleunpack code and that is verbose AF (to the point I made an earlier fix to stop it from telling me it had encountered some zeros, and then all 16k zeros it found!).

I hadn't realised that the pipe error might be overflow rather than disconnection. Does it make sense that the problem never resolves?

Your suggestion might well explain why it is sometimes hit but not always, as buffer filling is likely to be system load dependent.

Thanks for the pointer - I'll dig more and see what I can figure out.

Cheers,
Anthony

[chaley]

Originally Posted by stoduk

I hadn't realised that the pipe error might be overflow rather than disconnection. Does it make sense that the problem never resolves?

Standard outputs are pipes. What a process does when it fails to write to a pipe is up to the process. A process can report a "failure to write" when it really had a "failure to write within some delay." The process can fail or it can wait and try again. Dealing with transients is hard, so most people do the first: die.

TBH, I would be surprised if OSX fails in this way. Its underlying OS is based on BSD unix, and there is almost no chance that the underlying OS wouldn't handle throttling correctly. On the other hand, it is possible that the terminal application throttles by reporting errors instead of by waiting. I don't know.

[stoduk]

I can't produce the problem by throwing calls to print or job.log_write, which is odd, but perhaps it requires cpu or disk load at the same time. Removing all the guff that the plugin was spitting out (mostly from kindleunpack) seems to do the trick though.

So it looks likely you were right - thanks!

[kovidgoyal]

On OS X stdout and stderr are not the normal pipes to a tty.

They are instead redirected to asl which is how OS X display them in Console.app. See the setup_asl() function in the calibre source code. Since asl is a userspace daemon, it is quite possible that its failure modes are load dependant.

[stoduk]

Hi Kovid,

Thanks for the extra detail. Since silencing the plugin/its libraries I've not seen the problem again, so it seems very likely this is to blame.

I'm not entirely sure what the underlying problem is (if it genuinely a broken pipe does that mean ASL is closing the pipe if we send too much??).

Regardless, it is an easy enough problem to workaround - and hopefully having it documented here will help others.

Cheers,
Anthony