12-23-2015, 12:51 PM | #1 |
Member
Posts: 18
Karma: 10
Join Date: Sep 2013
Device: Kobo Mini, Kindle Paperwhite V2
|
Plugin error: IOError: [Errno 32] Broken pipe
Hi Folks,
I'm doing some development for a plugin (it is the X-Ray plugin that Matthew Wilson originally wrote), and I'm hitting a rather puzzling problem. Copious notes below, but basically writing to see if: - anyone is seeing similar issues with this or other plugins - anyone has any ideas where to look for the problem The symptom of the problem is simply the IOError(32, "Broken pipe") exception being raised when the plugin tries to write to stdout (directly or via print). I have found various mentions of this exception relating to Calibre, but a lot of those seem to be caused by virus software causing problems getting webpages. I found one mention of a different case, but no solution or further information to go on. Cheers, Anthony Sometimes I cannot reproduce the problem if I try (mostly when I've set aside some time to look at it!), the rest of the time I hit it very readily. The problem is never hit on the first run of the plugin, but is very often hit on the second and subsequent runs. It takes a restart of Calibre to get things working again. The error is seen when a call to print or sys.stdout.write is made in that plugin - and depending on which of those is called I either get the IOError(32, "Broken pipe") mentioned in the subject or sometimes get an odd IOError(0) exception. I've had a quick dig around the code to see where pipes are used - I'm assuming the problem is that whatever our plugin's stdout is connected to is has closed the pipe, hence the exception. However, "pipe" seems to be found a lot in the code - so really it seems more sensible to try to understand how the code works/where likely pipes are, rather than finding the bug via code inspection |
12-23-2015, 01:01 PM | #2 |
Grand Sorcerer
Posts: 11,939
Karma: 7219261
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
|
If you are using windows as your development environment then the problem is probably caused by windows not being able to buffer and write output to the screen as fast as the program can generate it. Simply redirect standard output and standard error to a file and the problem goes away. For example, when debugging calibre using print statements that generate lots of output I must use:
Code:
calibre-debug -g > foo.txt 2>&1 |
12-23-2015, 01:12 PM | #3 |
Member
Posts: 18
Karma: 10
Join Date: Sep 2013
Device: Kobo Mini, Kindle Paperwhite V2
|
OK, should have given more details Calibre 2.44.1 running on OS X 10.10.5.
However, your suggestion might still hold on OS X too - the plugin has borrowed the kindleunpack code and that is verbose AF (to the point I made an earlier fix to stop it from telling me it had encountered some zeros, and then all 16k zeros it found!). I hadn't realised that the pipe error might be overflow rather than disconnection. Does it make sense that the problem never resolves? Your suggestion might well explain why it is sometimes hit but not always, as buffer filling is likely to be system load dependent. Thanks for the pointer - I'll dig more and see what I can figure out. Cheers, Anthony |
12-23-2015, 02:45 PM | #4 | |
Grand Sorcerer
Posts: 11,939
Karma: 7219261
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
|
Quote:
TBH, I would be surprised if OSX fails in this way. Its underlying OS is based on BSD unix, and there is almost no chance that the underlying OS wouldn't handle throttling correctly. On the other hand, it is possible that the terminal application throttles by reporting errors instead of by waiting. I don't know. |
|
12-23-2015, 06:01 PM | #5 |
Member
Posts: 18
Karma: 10
Join Date: Sep 2013
Device: Kobo Mini, Kindle Paperwhite V2
|
I can't produce the problem by throwing calls to print or job.log_write, which is odd, but perhaps it requires cpu or disk load at the same time. Removing all the guff that the plugin was spitting out (mostly from kindleunpack) seems to do the trick though.
So it looks likely you were right - thanks! |
12-23-2015, 09:56 PM | #6 |
creator of calibre
Posts: 44,336
Karma: 23661992
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
On OS X stdout and stderr are not the normal pipes to a tty.
They are instead redirected to asl which is how OS X display them in Console.app. See the setup_asl() function in the calibre source code. Since asl is a userspace daemon, it is quite possible that its failure modes are load dependant. |
12-30-2015, 07:14 AM | #7 |
Member
Posts: 18
Karma: 10
Join Date: Sep 2013
Device: Kobo Mini, Kindle Paperwhite V2
|
Hi Kovid,
Thanks for the extra detail. Since silencing the plugin/its libraries I've not seen the problem again, so it seems very likely this is to blame. I'm not entirely sure what the underlying problem is (if it genuinely a broken pipe does that mean ASL is closing the pipe if we send too much??). Regardless, it is an easy enough problem to workaround - and hopefully having it documented here will help others. Cheers, Anthony |
Tags |
plugin development |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
ERROR: Unhandled exception: <b>IOError</b>:[Errno 32] Broken pipe | Martin77 | Calibre | 3 | 12-24-2015 07:50 AM |
IOError: [Errno 22] invalid mode ('w') or filename | JessRawesome | Calibre | 6 | 02-17-2015 09:15 PM |
IOError: [Errno 22] Invalid argument | jkcoder | Calibre | 5 | 09-25-2010 09:28 AM |
IOError: [Errno 13] Permission denied | s3ntient | Calibre | 3 | 02-20-2010 02:01 PM |
IOError errno 5? | codemac | Calibre | 11 | 10-22-2009 02:36 PM |