Terminate process in ThreadedJob terminates calibre main process

[xxyzz]

I was using spaCy's multiprocessing pipe(https://github.com/explosion/spaCy/b...guage.py#L1581) in ThreadedJob but found it will terminate calibre, I think the code should looks like this:

from multiprocessing import Process
from threading import Thread

def fun():
    while True:
        pass

class JobThread(Thread):
    def run(self):
        # code in ThreadedJob
        p = Process(target=fun)
        p.start()
        p.terminate()

JobThread().start()  # pretend it's ThreadedJobServer

The process created in ThreadedJob is still running but the calibre main process is terminated.

I also want to know the difference between calibre's ParallelJob and ThreadedJob.

[kovidgoyal]

What is it yu are actually trying to do?

[xxyzz]

I just want to try spaCy's multiprocessing pipe feature, and I put run the code inside ThreadJob to prevent these code to freeze calibre. If enable spacy's multiprocessing mode, it will create some processes run a `while` loop then terminate them when they are done. But these processes are still running and calibre is terminated instead.

I don't understand why sending SIGTERM to these processes somehow terminates calibre.

[kovidgoyal]

Presumably it is sending SIGTERM to the main calibre process instead.

[xxyzz]

But shouldn't `p.terminate()` send the signal just to `p`?

[kovidgoyal]

You cannot use multiprocessing safely in native code programs (such as calibre) that embed a python interpreter. In fact, you cant use it safely at all, it is one of the worst designed modules in the standard library. Read its docs, in particular

The 'spawn' and 'forkserver' start methods cannot currently be used with “frozen” executables (i.e., binaries produced by packages like PyInstaller and cx_Freeze) on Unix. The 'fork' start method does work.

And fork is broken in any program that uses threading anyway. So multiprocessing is a complete no go.

Again, let me ask, what it is you want to actually do. calibre already has the ability to run worker processes in several ways. It does not however touch multiprocessing with ten foot barge pole.

[xxyzz]

I'm using spacy's `nlp.pipe()` to find named entities in the book text, this function has a parameter 'n_process' to set how many processes to run this function. The default value is 1 and no process is created. I want to see whether enable multiprocessing will be faster.

Oh, I don't know fork a multithreaded process is problematic as the doc says.

And could you please explain the difference between ParallelJob and ThreadedJob?

[kovidgoyal]

ParallelJob is designed to work with pre-defined entry points in ipc/worker.py that are run in worker processes. ThreadedJob is meant for in-process jobs.

However, either of these classes are meant to be used only with calibre's jobs system (the jobs button in the main calibre gui). If you want to do jobs in multiple processes as part of larger job, your best bet is to use the ipc/simple_worker.py module. See for example how it is used in srv/render_book.py where calibre uses multiple worker processes to transform a book into a form suitable for the calibre viewer.