|
I've all post processing options turned off (nothing checked, calculate word count set to No), other wise the update would take closer to an hour than 20 minutes.
I do realize that the stories I'm following are really large, but the low performance doesn't come from the site access.
If I don't use the "background metadata" option, then fetching metadata for 45 huge books takes about 2 minutes, which isn't too bad.
And if there are multiple new chapters, I can see from the job log that the chapters are downloading at a rate of about 1 chapter per second, so that's pretty good too.
It's basically the loading and saving of the epub that takes significant time as far as I can tell (no surprise there, they are huge).
Maybe schedule the individual books as individual jobs, but have a global list with domain name + lock (whatever that is in python) and acquire the lock while loading a page from that domain.
This would serialize the online access (hopefully preventing a possible ban as you mentioned) while still allowing local processing to take place in parallel.
Anyway, just an idea. FFF as is has saved me countless hours of manual work and I really appreciate the work you've put in.
|