[chaley]

Originally Posted by davidfor

This is probably going to be a pain to work out. The only thing I can think of is that a separate thread is being started to do the work for each file. Something is happening that either one is failing or getting "lost". There have been some issues on Macs with the thread management with some occasional crashes. This could be a less obvious symptom of that.

When it happens, there might be an error in the calibre debug log. But, to see it, you would need to run calibre in debug mode and check the log. Or keep it until you see a book with the problem. Either way it will be painful.

1. I don't think this will help, but one thing you might try is to tell the thread pool to use fewer threads. By default it uses 'number_of_processors * 5'. You can reduce that by giving "max_workers=X" (for some X) in "with ThreadPoolExecutor() as pool" in container.py.__run_async().
2. Another thought. __run_async() waits for a maximum of 10 seconds for each threaded task submitted to the pool. It waits in task submission order. Even if any task running by itself is guaranteed to finish in 10 seconds, it is very possible that a task early in the queue is starved by other tasks and violates the timeout. This can also happen if the machine is busy. This raises two questions:
   • Why 10 seconds? Why not unlimited (None)? Why not 100 seconds?
   • It isn't obvious what happens if finish() raises TimeoutError, which it will do when the timeout is hit.
     • It seems that finish() won't be called for the remaining tasks, leaving their threads waiting to terminate.
     • It seems that the exception will eventually be caught and logged, but what processing is skipped is hard (for me) to see.
3. In any event, after looking at the docs I don't think the code is right. The python docs suggests a different structure for the code. It wants the calls to finish() to be inside the "with" statement, which makes sense because the thread pool will be cancelled when the "with ..." statement completes. It also suggests that the finish() call be wrapped in a try/except block.
   
   This doc is also good, but long and somewhat hard to read. It suggests that your case should look something like the Submit and Use as Completed pattern. Here is an example. I left the 10 second timeout, but as before I am not sure why it is there. A TimeoutError is raised if 10 seconds pass with no task finishing.
   

       from concurrent.futures import as_completed
       from concurrent.futures import TimeoutError
       with ThreadPoolExecutor() as pool:
           futures = [pool.submit(func, *arg) for arg in args]
           try:
               for future in as_completed(futures, timeout=10):
                   try:
                       future.result()
                   except Exception as e:
                       print(f'Exception {str(e)} waiting for task to finish')
           except TimeoutError as t:
               print('A task took too long')