[GUI Plugin] FanFicFare

[Lys]

Originally Posted by JimmXinu

You could try use_archiveofourown_gay:true or Browser Cache feature. See wiki page for ProblematicSites

And worst case, AO3 has their own epub download feature you can use.

I tried different combinations with use_archiveofourown_gay:true or with the browser cache feature, but the result was always the same (with a slight different long message error, but always a 403)

That's the reason I asked if someone managed to download locked AO3 fics through FFF in the last few days, and if yes, if they could share their own setting to see if I could replicate that and solve the issue

[ownedbycats]

Originally Posted by Lys

Hi all

I'm having some issues updating any locked story from AO3. Not locked stories are updating correctly, instead.
I tried playing around with my personal ini (taking out always_login + credential, adding the open in browser one), but no matter what I did, I always got a 403 error.
Are you also experiencing this issue?
If yes, and you managed to solve it, can you please share your personal ini setting?

Originally Posted by Lys

I tried different combinations with use_archiveofourown_gay:true or with the browser cache feature, but the result was always the same (with a slight different long message error, but always a 403)

That's the reason I asked if someone managed to download locked AO3 fics through FFF in the last few days, and if yes, if they could share their own setting to see if I could replicate that and solve the issue

Restricted stories work as expected for me.

How long a period are you testing over? I get occasional 525 errors, but they clear up the next day. I suspect related to repeated requests when they're getting higher traffic.

Might also be worth rebooting your modem in case you got assigned an IP used by a previous bad actor.

[Lys]

Originally Posted by ownedbycats

Restricted stories work as expected for me.

How long a period are you testing over? I get occasional 525 errors, but they clear up the next day. I suspect related to repeated requests when they're getting higher traffic.

Might also be worth rebooting your modem in case you got assigned an IP used by a previous bad actor.

I came back from holiday after a couple of weeks, so... since yesterday. Two weeks ago, everything was fine

Initially, I had a 403 error for everything (locked and not locked stories), then I commented always_login:true and my username, and not locked stories could be downloaded again.
For locked stories, instead, no luck so far, no matter which instruction I try to comment or to add: I always get a 403 or a Failed to Login for URL

I'll try rebooting my modem and see how it goes!

[JimmXinu]

It's not just a matter of settings. Cloudflare can and does give different treatment to different people on different networks.

I'm currently able to download 'locked' AO3 stories using FFF's automatic login.

[wildfire070]

Originally Posted by Lys

I tried different combinations with use_archiveofourown_gay:true or with the browser cache feature, but the result was always the same (with a slight different long message error, but always a 403)

That's the reason I asked if someone managed to download locked AO3 fics through FFF in the last few days, and if yes, if they could share their own setting to see if I could replicate that and solve the issue

I'm not having any issues downloading locked/unlocked fics. I have this in my personal.ini under the AO3 section:

use_archiveofourown_gay:true
always_login:true
username:
password:
slow_down_sleep_time:4
use_view_full_work:true

[DevilsReject97]

Received this error with FFF today, never seen this one before:

AttributeError: 'NoneType' object has no attribute 'find'

Basic settings aside from adding the setup for flaresolverr

[JimmXinu]

Originally Posted by DevilsReject97

Received this error with FFF today, never seen this one before:

AttributeError: 'NoneType' object has no attribute 'find'

Basic settings aside from adding the setup for flaresolverr

I can't even look at this without a story URL and/or debug log.

[DevilsReject97]

Originally Posted by JimmXinu

I can't even look at this without a story URL and/or debug log.

My apologies.

The link was https://novelfull.com/the-genius-sys...out-equal.html

I also got it on a second story, but it was the same site.

I got it twice on the same story, but now the story will run some and then cap out as a timeout on Flaresolverr.... which seems odd because it runs to about 7%

[DevilsReject97]

I honestly am suspecting its the site being difficult and not anything to do with FFF - I retried for a 3rd time and it's already doubled what it previously did running.

As a sidenote, if I get a failed challenge response for flaresolverr within FFF, is there a setting I should change?

[DevilsReject97]

Originally Posted by DevilsReject97

I honestly am suspecting its the site being difficult and not anything to do with FFF - I retried for a 3rd time and it's already doubled what it previously did running.

As a sidenote, if I get a failed challenge response for flaresolverr within FFF, is there a setting I should change?

Spoke too soon...

I just added the last bits of the log as it was working up until Ch. 285.

[JimmXinu]

If you're downloading hundreds of chapters, and it's not failing at the same place, I strongly suspect you are running into some form of blocking from cloudflare or the site.

(Geez, a story with 850 chapters and 18 pages just of ToC... I downloaded the first 5 chapters without issue.)

You can try:

• Increasing slow_down_sleep_time: (default is 5 for novelfull.com)
• Only downloading a hundred or so chapters at a time using story URL with chapter range. Ie, https://novelfull.com/the-genius-sys...out-equal.html[1-100], then when that finishes, wait a while, then update by story URL with [101-200], etc.
• Set continue_on_chapter_error:true -- not necessary a good idea if you don't also do ranges, because FFF will continue trying for the next 600 chapters of failures if you give it the whole thing and it starts failing.

The advantage of continue_on_chapter_error:true is that FFF will keep all the chapters already downloaded. The downside is that continuing to try while failing may get you blocked longer.

Maybe I will investigate adding an intermediate setting that keeps chapters that have succeeded, but still stops on first failure.

[DevilsReject97]

Originally Posted by JimmXinu

If you're downloading hundreds of chapters, and it's not failing at the same place, I strongly suspect you are running into some form of blocking from cloudflare or the site.

(Geez, a story with 850 chapters and 18 pages just of ToC... I downloaded the first 5 chapters without issue.)

You can try:

• Increasing slow_down_sleep_time: (default is 5 for novelfull.com)
• Only downloading a hundred or so chapters at a time using story URL with chapter range. Ie, https://novelfull.com/the-genius-sys...out-equal.html[1-100], then when that finishes, wait a while, then update by story URL with [101-200], etc.
• Set continue_on_chapter_error:true -- not necessary a good idea if you don't also do ranges, because FFF will continue trying for the next 600 chapters of failures if you give it the whole thing and it starts failing.

The advantage of continue_on_chapter_error:true is that FFF will keep all the chapters already downloaded. The downside is that continuing to try while failing may get you blocked longer.

Maybe I will investigate adding an intermediate setting that keeps chapters that have succeeded, but still stops on first failure.

This is what I have for my settings:

Default ini for FFF and this:
[novelfull.com]
use_flaresolverr_proxy:true
slow_down_sleep_time:7
## flaresolverr_proxy_timeout is in integer milliseconds.
## Keep flaresolverr_proxy_timeout set smaller than connect_timeout
## otherwise the connection *to* flaresolverr will timeout first.
flaresolverr_proxy_timeout:59000

If 5 is better, or 10 is better?


Thank you for looking into it. I very much appreciate the help as I really only read books from this site because most of the others that are similar don't work with FFF. I can certainly try to do that, as updates may be easier. I've actually wondered the same, as all 3 attempts that downloaded all stopped at different spots with the same error.

Ironically, the site itself has been acting weird even with updates, as I have books with over 3k chapters but FFF will say the site only has 1100 chapters available - yet the site shows otherwise.

Most likely, it is as you suspect and the site itself will probably become more unusable in this manner.

[JimmXinu]

Originally Posted by JimmXinu

The advantage of continue_on_chapter_error:true is that FFF will keep all the chapters already downloaded. The downside is that continuing to try while failing may get you blocked longer.

Maybe I will investigate adding an intermediate setting that keeps chapters that have succeeded, but still stops on first failure.

When continue_on_chapter_error was added, the use case was for old SB/SV story threads with manually crafted ToC posts that sometimes contained errors.

Today, we're seeing more and more sites that appear to start blocking after some unknown access limit is exceeded.

continue_on_chapter_error is helpful in those cases because you don't discard however many chapters did download before getting blocked. But I suspect that continuing to try each subsequent chapter after blocking starts will increase your blocked time--that's how I implemented it back in the day.

I propose to add a second setting: continue_on_chapter_error_try_limit.

FFF, (with continue_on_chapter_error:true), would record each chapter error and continue to the next chapter, but after continue_on_chapter_error_try_limit failures, would give up and mark all subsequent chapters 'ERROR' without trying to download them.

I'm thinking an appropriate default would be continue_on_chapter_error_try_limit:5

Some details:

• Why not just leave out the failed / not-tried chapters?
  Because users won't immediately notice missing chapters--I did exactly that while testing that very thing.
  
• What happens with updates?
  That's the real power of continue_on_chapter_error in this case; FFF will discard the ERROR message chapters and attempt to download them as if they were newly added. (Including be marked (new) if mark_new_chapters:true.)
  You could, for example, update a story with 1000 chapters once a day and if you get 100 or 200 chapters each time, have the whole thing in a week or so. Previously, we've instructed users to basically do that manually with chapter ranges.
  
• What about the original use case? What if there are actually that many bad URLs? How can you ever get the chapters after those?
  You'd need to increase continue_on_chapter_error_try_limit for that story, or use ignore_chapter_url_list.
  
• The ERROR chapter title would be the same, but ERROR chapter text for chapters after continue_on_chapter_error_try_limit would have different explanatory text.
  
• 'Skipping subsequent chapters without trying' does not apply to later chapters that were already in the epub. Those will still be used.

Opinions?

[DevilsReject97]

Originally Posted by JimmXinu

When continue_on_chapter_error was added, the use case was for old SB/SV story threads with manually crafted ToC posts that sometimes contained errors.

Today, we're seeing more and more sites that appear to start blocking after some unknown access limit is exceeded.

continue_on_chapter_error is helpful in those cases because you don't discard however many chapters did download before getting blocked. But I suspect that continuing to try each subsequent chapter after blocking starts will increase your blocked time--that's how I implemented it back in the day.

I propose to add a second setting: continue_on_chapter_error_try_limit.

FFF, (with continue_on_chapter_error:true), would record each chapter error and continue to the next chapter, but after continue_on_chapter_error_try_limit failures, would give up and mark all subsequent chapters 'ERROR' without trying to download them.

I'm thinking an appropriate default would be continue_on_chapter_error_try_limit:5

Some details:

• Why not just leave out the failed / not-tried chapters?
  Because users won't immediately notice missing chapters--I did exactly that while testing that very thing.
  
• What happens with updates?
  That's the real power of continue_on_chapter_error in this case; FFF will discard the ERROR message chapters and attempt to download them as if they were newly added. (Including be marked (new) if mark_new_chapters:true.)
  You could, for example, update a story with 1000 chapters once a day and if you get 100 or 200 chapters each time, have the whole thing in a week or so. Previously, we've instructed users to basically do that manually with chapter ranges.
  
• What about the original use case? What if there are actually that many bad URLs? How can you ever get the chapters after those?
  You'd need to increase continue_on_chapter_error_try_limit for that story, or use ignore_chapter_url_list.
  
• The ERROR chapter title would be the same, but ERROR chapter text for chapters after continue_on_chapter_error_try_limit would have different explanatory text.
  
• 'Skipping subsequent chapters without trying' does not apply to later chapters that were already in the epub. Those will still be used.

Opinions?

I like the concept, because I was trying your do the 100 chapters thing and I was still getting similar issues with the AttributeError: 'NoneType' object has no attribute 'find' after so many chapters would succeed.

Originally Posted by JimmXinu

Maybe I will investigate adding an intermediate setting that keeps chapters that have succeeded, but still stops on first failure.

Would this be something that could be tied into your other suggestion? I think with this, it would in fact resolve at least all of my issues, and probably many others...

[culytera]

Originally Posted by indecisivemess

<snipped chunk of personal.ini>

I couldn't make heads nor tails of what's going wrong with the code you shared, so I copied mine over and adjusted some lines to work with your setup, but it's possible I missed some parts. I had to download a portable build of calibre and copy your custom columns so I can do some tests. I wrote a few notes on the personal.ini you shared and the changes I made:

- Why was the [default] section label commented?
- Removed css-style comments in the add_to_output_css section, Fanficfare only accepts # for comments and in their own lines
- Remove inline comments in metadata replacements, affects metadata values and how the lines are parsed
- Used double hashtags for commentary and single hashtag for commenting an option
- Added another replace_metadata line for icons since you have always_login set to false which would mean empty values for bookmarked and subscribed, this will set the default icon to what it would also be if not bookmarked and not subscribed
- Changed to numChapters=>#chapters since the #chapters column is an integer type

The icons are in HTML code so I have to share it from a pastebin site or else mobileread will convert it into an icon.

Attached are the Fanficfare settings you'll need to adjust.