View Single Post
Old 06-22-2011, 02:29 PM   #13
chaley
Grand Sorcerer
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 12,452
Karma: 8012886
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
Quote:
Originally Posted by kovidgoyal View Post
@chaley: A pipe sounds interesting, I've never implemented one, I guess it is basically a FIFO Queue that only reads a few KB at a time into the queue. The question is what is the best architecture for filling the buffer? A separate thread or just fill when emptied in the emptying thread? The tradeoff, I guess, is between amount of time file handle has to be help open vs. efficiency?
Pipes are indeed fifo queues, and are very easy to use. os.pipe returns the two file descriptors that act just as you would expect.

The best way to write to pipes is to use a thread. Normally, pipe writing will block until the reader takes data, and this can be a pain to work around, usually requiring the use of "select". I don't know if python has a good/usable implementation of select.

If you use a thread, then both open file handling and feeding the pipe are natural, using a "with ...:" construction. The same thing can support locks by using a "with" inside a "with".
chaley is offline   Reply With Quote