Quote:
Originally Posted by JimmXinu
When continue_on_chapter_error was added, the use case was for old SB/SV story threads with manually crafted ToC posts that sometimes contained errors.
Today, we're seeing more and more sites that appear to start blocking after some unknown access limit is exceeded.
continue_on_chapter_error is helpful in those cases because you don't discard however many chapters did download before getting blocked. But I suspect that continuing to try each subsequent chapter after blocking starts will increase your blocked time--that's how I implemented it back in the day.
I propose to add a second setting: continue_on_chapter_error_try_limit.
FFF, (with continue_on_chapter_error:true), would record each chapter error and continue to the next chapter, but after continue_on_chapter_error_try_limit failures, would give up and mark all subsequent chapters 'ERROR' without trying to download them.
I'm thinking an appropriate default would be continue_on_chapter_error_try_limit:5
Some details:
- Why not just leave out the failed / not-tried chapters?
Because users won't immediately notice missing chapters--I did exactly that while testing that very thing.
- What happens with updates?
That's the real power of continue_on_chapter_error in this case; FFF will discard the ERROR message chapters and attempt to download them as if they were newly added. (Including be marked (new) if mark_new_chapters:true.)
You could, for example, update a story with 1000 chapters once a day and if you get 100 or 200 chapters each time, have the whole thing in a week or so. Previously, we've instructed users to basically do that manually with chapter ranges.
- What about the original use case? What if there are actually that many bad URLs? How can you ever get the chapters after those?
You'd need to increase continue_on_chapter_error_try_limit for that story, or use ignore_chapter_url_list.
- The ERROR chapter title would be the same, but ERROR chapter text for chapters after continue_on_chapter_error_try_limit would have different explanatory text.
- 'Skipping subsequent chapters without trying' does not apply to later chapters that were already in the epub. Those will still be used.
Opinions? |
I like the concept, because I was trying your do the 100 chapters thing and I was still getting similar issues with the AttributeError: 'NoneType' object has no attribute 'find' after so many chapters would succeed.
Quote:
Originally Posted by JimmXinu
Maybe I will investigate adding an intermediate setting that keeps chapters that have succeeded, but still stops on first failure.  |
Would this be something that could be tied into your other suggestion? I think with this, it would in fact resolve at least all of my issues, and probably many others...