Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Development

Notices

Reply
 
Thread Tools Search this Thread
Old 12-23-2015, 12:51 PM   #1
stoduk
Member
stoduk began at the beginning.
 
Posts: 18
Karma: 10
Join Date: Sep 2013
Device: Kobo Mini, Kindle Paperwhite V2
Plugin error: IOError: [Errno 32] Broken pipe

Hi Folks,

I'm doing some development for a plugin (it is the X-Ray plugin that Matthew Wilson originally wrote), and I'm hitting a rather puzzling problem.

Copious notes below, but basically writing to see if:
- anyone is seeing similar issues with this or other plugins
- anyone has any ideas where to look for the problem

The symptom of the problem is simply the IOError(32, "Broken pipe") exception being raised when the plugin tries to write to stdout (directly or via print).

I have found various mentions of this exception relating to Calibre, but a lot of those seem to be caused by virus software causing problems getting webpages. I found one mention of a different case, but no solution or further information to go on.

Cheers,
Anthony

Sometimes I cannot reproduce the problem if I try (mostly when I've set aside some time to look at it!), the rest of the time I hit it very readily. The problem is never hit on the first run of the plugin, but is very often hit on the second and subsequent runs. It takes a restart of Calibre to get things working again.

The error is seen when a call to print or sys.stdout.write is made in that plugin - and depending on which of those is called I either get the IOError(32, "Broken pipe") mentioned in the subject or sometimes get an odd IOError(0) exception.

I've had a quick dig around the code to see where pipes are used - I'm assuming the problem is that whatever our plugin's stdout is connected to is has closed the pipe, hence the exception. However, "pipe" seems to be found a lot in the code - so really it seems more sensible to try to understand how the code works/where likely pipes are, rather than finding the bug via code inspection
stoduk is offline   Reply With Quote
Old 12-23-2015, 01:01 PM   #2
chaley
Grand Sorcerer
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 11,795
Karma: 7029971
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
If you are using windows as your development environment then the problem is probably caused by windows not being able to buffer and write output to the screen as fast as the program can generate it. Simply redirect standard output and standard error to a file and the problem goes away. For example, when debugging calibre using print statements that generate lots of output I must use:
Code:
calibre-debug -g > foo.txt 2>&1
chaley is offline   Reply With Quote
Advert
Old 12-23-2015, 01:12 PM   #3
stoduk
Member
stoduk began at the beginning.
 
Posts: 18
Karma: 10
Join Date: Sep 2013
Device: Kobo Mini, Kindle Paperwhite V2
OK, should have given more details Calibre 2.44.1 running on OS X 10.10.5.

However, your suggestion might still hold on OS X too - the plugin has borrowed the kindleunpack code and that is verbose AF (to the point I made an earlier fix to stop it from telling me it had encountered some zeros, and then all 16k zeros it found!).

I hadn't realised that the pipe error might be overflow rather than disconnection. Does it make sense that the problem never resolves?

Your suggestion might well explain why it is sometimes hit but not always, as buffer filling is likely to be system load dependent.

Thanks for the pointer - I'll dig more and see what I can figure out.

Cheers,
Anthony
stoduk is offline   Reply With Quote
Old 12-23-2015, 02:45 PM   #4
chaley
Grand Sorcerer
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 11,795
Karma: 7029971
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
Quote:
Originally Posted by stoduk View Post
I hadn't realised that the pipe error might be overflow rather than disconnection. Does it make sense that the problem never resolves?
Standard outputs are pipes. What a process does when it fails to write to a pipe is up to the process. A process can report a "failure to write" when it really had a "failure to write within some delay." The process can fail or it can wait and try again. Dealing with transients is hard, so most people do the first: die.

TBH, I would be surprised if OSX fails in this way. Its underlying OS is based on BSD unix, and there is almost no chance that the underlying OS wouldn't handle throttling correctly. On the other hand, it is possible that the terminal application throttles by reporting errors instead of by waiting. I don't know.
chaley is offline   Reply With Quote
Old 12-23-2015, 06:01 PM   #5
stoduk
Member
stoduk began at the beginning.
 
Posts: 18
Karma: 10
Join Date: Sep 2013
Device: Kobo Mini, Kindle Paperwhite V2
I can't produce the problem by throwing calls to print or job.log_write, which is odd, but perhaps it requires cpu or disk load at the same time. Removing all the guff that the plugin was spitting out (mostly from kindleunpack) seems to do the trick though.

So it looks likely you were right - thanks!
stoduk is offline   Reply With Quote
Advert
Old 12-23-2015, 09:56 PM   #6
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 44,028
Karma: 22669822
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
On OS X stdout and stderr are not the normal pipes to a tty.

They are instead redirected to asl which is how OS X display them in Console.app. See the setup_asl() function in the calibre source code. Since asl is a userspace daemon, it is quite possible that its failure modes are load dependant.
kovidgoyal is offline   Reply With Quote
Old 12-30-2015, 07:14 AM   #7
stoduk
Member
stoduk began at the beginning.
 
Posts: 18
Karma: 10
Join Date: Sep 2013
Device: Kobo Mini, Kindle Paperwhite V2
Hi Kovid,

Thanks for the extra detail. Since silencing the plugin/its libraries I've not seen the problem again, so it seems very likely this is to blame.

I'm not entirely sure what the underlying problem is (if it genuinely a broken pipe does that mean ASL is closing the pipe if we send too much??).

Regardless, it is an easy enough problem to workaround - and hopefully having it documented here will help others.

Cheers,
Anthony
stoduk is offline   Reply With Quote
Reply

Tags
plugin development

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
ERROR: Unhandled exception: <b>IOError</b>:[Errno 32] Broken pipe Martin77 Calibre 3 12-24-2015 07:50 AM
IOError: [Errno 22] invalid mode ('w') or filename JessRawesome Calibre 6 02-17-2015 09:15 PM
IOError: [Errno 22] Invalid argument jkcoder Calibre 5 09-25-2010 09:28 AM
IOError: [Errno 13] Permission denied s3ntient Calibre 3 02-20-2010 02:01 PM
IOError errno 5? codemac Calibre 11 10-22-2009 02:36 PM


All times are GMT -4. The time now is 07:48 PM.


MobileRead.com is a privately owned, operated and funded community.