Hi - the Count Pages plugin has gone into a meltdown ever since I foolishly thought I could trivially split its execution into multiple jobs instead of one big job.
The point of running them in smaller jobs was that if a user decided to count pages on 1000 books, then realised they had to turn their computer off, at least with a default batch size of 50 then they would have got "some" of their books updated by that point, rather than all the results so far being lost.
The problem I am seeing in hindsight is that Calibre is running multiple of those jobs in parallel. Which all starts off fine initially until the jobs start spitting out errors like this:
Code:
Traceback (most recent call last):
File "calibre_plugins.count_pages.jobs", line 191, in do_statistics_for_book
File "calibre_plugins.count_pages.statistics", line 72, in get_page_count
File "calibre_plugins.count_pages.statistics", line 114, in _open_epub_file
File "calibre\ebooks\oeb\iterator\book.py", line 147, in __enter__
File "calibre\ebooks\oeb\iterator\book.py", line 87, in run_extract_book
File "calibre\utils\ipc\simple_worker.py", line 251, in fork_job
File "calibre\utils\ipc\simple_worker.py", line 176, in run_job
File "calibre\utils\ipc\simple_worker.py", line 119, in communicate
calibre.utils.ipc.simple_worker.WorkerError: Worker failed
AI tells me that it thinks it is resource exhaustion of some kind, but hasn't managed a solution for it. It was saying it thought the problem was this line:
Code:
server = Server(pool_size=cpus)
but then even setting the pool size to 1 it still has a problem (not that I know what difference it is making).
Does Kovid or anyone who has used the jobs stuff have any suggestions? The code is right there if you want to experiment with the latest release. If I get no joy with it I will just revert the feature. I am so out of touch with the calibre code and no time to try to properly learn it, most definitely this code would have been copied from calibre somewhere about 15 years ago!