Quote:
Originally Posted by kovidgoyal
The problem with threads and sockets is performance, there's no select, only poll for each socket. This will be rather nasty in the thread that manages the connections from child processes, unless we launch a new thread per child. And given that only a single python thread can run at a time...
|
I don't think the GIL is really an issue. Python will switch between the threads as it wishes, distributing the time to do the copy over several context switches. Yes, multiple cores are not used, but this is true even if the file is copied. The work gets done in the same amount of time, but threaded solutions better spread that work over time.
Quote:
|
Given all these complications, to me it's just more reasonable to export in process ->work out of process -> import in process rather than try to do everything out of process. The cost is one extra copy per file. Which given the file sizes of typical books seems fairly reasonable to me.
|
If the problem is not producer-consumer, then you are absolutely correct. If it is P/C, then I am not convinced.
One thing I don't know is what percentage of operations are what. Readers are P/C, but modifiers are not.