New Test Version Posted
2021-01-28
- Plugin: Only run one BG download at time.
- Get cloudscraper 1.2.56 changes, but keep py2 compat.
Some users over on the mailing list pointed out running parallel downloads on the same site is more likely to trip blocking. This version prevents that by only running one download at a time.
I said over there:
Quote:
So, I've been researching this a bit more.
First, FFF plugin launches a background job in Calibre. As V1cT1T10uS pointed out, Calibre has a setting for how many BG jobs to run at a time--often set (or limited) to the number of CPUs.
The weird part is that FFF has *also* been launching that many parallel process again from its BG process. So if you launched FFF with a bunch of downloads *and* started a bunch of converts or something, Calibre could technically be running twice the configured number of background processes (but not jobs in the job list).
Way back when, I lifted the BG processing code for FFF plugin from the ISBN plugin and I honestly didn't understand exactly what was going on at the time. The inherited comment even says that it launched a process per book to work around a memory leak issue in ISBN.
I've posted a test version plugin that only does one download at a time:
https://www.mobileread.com/forums/sh...25&postcount=2
Currently, it still launches a separate process for each download, but I will be experimenting with what can be done--all in one process (like CLI does lists), maybe process per site.
|