[GUI Plugin] FanFicFare

[booster-t]

Well, FF.net is at again. The latest plugin reports bad chapters and doesn't download. Calibre 5.10.1 and plugin 3.99.27 yield this:

Traceback (most recent call last):
cloudscraper.exceptions.CloudflareChallengeError: Detected a Cloudflare version 2 challenge, This feature is not available in the opensource (free) version.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
calibre_plugins.fanficfare_plugin.fanficfare.excep tions.FailedToDownload: cloudscraper reports: "Detected a Cloudflare version 2 challenge, This

Story URL: https://www.fanfiction.net/s/1358165...rmione-Granger

This was new to me so nothing to overwite.

[ownedbycats]

Use the cache in the test version.

[watani]

Is there some sort of functionality in FFF that allows for a limited number of retries for failed chapter downloads? I've had several failed downloads that have occurred without any clear pattern due to some websocket issue that causes one chapter to fail out of the entire story. While an immediate retry successfully downloads the full thing, it'd be nice if you could tell FFF to retry the chapter download after a brief pause to not require fully querying the entire story again.

Additionally, it'd also be nice if there was some sort of option to only download new chapters rather than the whole fic especially given the restrictions with FF.net. I just attempted a download of a story with 305 chapters using the default sleep time settings (total download time was 1+ hours) and apparently it was enough to trigger the Cloudflare Lvl 2 challenge, preventing the final three chapters from being retrieved.

Edit: Apparently the triggering of the level 2 challenge may have been unrelated to my download, given booster-t's reply above?

Edit 2: Ignore my requests, apparently this functionality is already in FFF under a specific update mode and I somehow misunderstood the modes even after reading through the wiki.

[ownedbycats]

continue_on_chapter_error:true will add a placeholder page for failed chapters. I think it's become default in one of the new test versions, and update ePub will also try to re-fetch the failed chapters was also recently added.

I've been getting the Cloudflare challenges very sporadically too—I suspect ffnet might've misconfigured something.

[watani]

Originally Posted by ownedbycats

continue_on_chapter_error:true will add a placeholder page for failed chapters. I think it's become default in one of the new test versions, and update ePub will also try to re-fetch the failed chapters.

I've been getting the Cloudflare challenges very sporadically too—I suspect ffnet might've misconfigured something.

I know about the continue_on_chapter_error option since I'm using the latest test version, but what I'm hoping can be added to FFF is for an option to (within the original download attempt) retry with a limited number of attempts the retrieval of just the chapters it failed to download rather than just inserting placeholders. I don't want FFF to waste time updating the whole ePub if the other chapters were already successfully updated.

[ILB]

Originally Posted by watani

Additionally, it'd also be nice if there was some sort of option to only download new chapters rather than the whole fic especially given the restrictions with FF.net. I just attempted a download of a story with 305 chapters using the default sleep time settings (total download time was 1+ hours) and apparently it was enough to trigger the Cloudflare Lvl 2 challenge, preventing the final three chapters from being retrieved.

Which update mode do you use?
Change it to "Update EPUB if New Chapters" - but that´s only possible if you use EPUB format.

see also https://github.com/JimmXinu/FanFicFare/wiki/UpdateModes

Or you can tell FFF exactly which chapter it should download.
Add [1,5] after the URL to limit the download to chapters 1-5. Or whatever chapters you want .

[watani]

Originally Posted by ILB

Which update mode do you use?
Change it to "Update EPUB if New Chapters" - but that´s only possible if you use EPUB format.

see also https://github.com/JimmXinu/FanFicFare/wiki/UpdateModes

Ah, I see! My bad, I completely misunderstood the purpose of those modes since I thought that they still updated the full ePub. If those modes only update with the new chapters that's exactly what I wanted.

[ILB]

Originally Posted by watani

Ah, I see! My bad, I completely misunderstood the purpose of those modes since I thought that they still updated the full ePub. If those modes only update with the new chapters that's exactly what I wanted.

That would be a perfect opportunity to tell JimmXinu if his documentation is understandable:

Originally Posted by JimmXinu

Someone took me to task a few weeks ago for the lack of documentation about what the different update modes mean in FFF plugin.

I have written a document to try and explain the basics:

https://github.com/JimmXinu/FanFicFare/wiki/UpdateModes

I'd appreciate it if people would review and comment on whether they think it is adequately clear and where it isn't.

I think it´s quite clear formulated but I use FFF for a long time. So I did already know most of it.

[JimmXinu]

Originally Posted by ownedbycats

I add the AO3 feeds for a few sections I like to my Mozilla Thunderbird's RSS reader to keep track of new stories. Thunderbird parses them as a text/html email, so I was able to drag & drop the ones I was interested in into the window. Now it just drops the .eml file.

I suspect that the recent change to fix notification emails with non-ascii (that you also asked for), because it now parses the message more as an email instead of an undifferentiated text block, did that.

If you want to attach some .eml files (ideally some with accented characters too) that this happens to, I will look at them. But RSS is not something I officially support.

[ownedbycats]

Originally Posted by JimmXinu

I suspect that the recent change to fix notification emails with non-ascii (that you also asked for), because it now parses the message more as an email instead of an undifferentiated text block, did that.

If you want to attach some .eml files (ideally some with accented characters too) that this happens to, I will look at them. But RSS is not something I officially support.

Here are two, one from ffnet and one from ao3. (Note that dragging and dropping them from a file manager results in the same as dropping from Thunderbird.) They're, as far as I can tell, valid RFC 5322 emls - Thunderbird converts the feed content into Content-Type: text/html; charset=UTF-8 messages.

One oddity though: the fic links aren't in the body itself, but in a non-standard Content-Base header, and editing it so the link is in the body itself causes it to accept. Is it possible that the changes that were made caused it to ignore headers?

I couldn't find any RSS text/html messages with accented characters, though the AO3 one contains a U+2026 ellipsis. But for testing purposes opening either file in text editor and adding the characters should work: ÀÉÏÕÛąĕĩőū

EDIT: It would be helpful if I attached the file.

[JimmXinu]

New Test Version Posted

2021-02-11
- Allow tweak_fg_sleep etc with any site, add defaults settings for basexenforo, fictionalley and fictionpress.
- Drag/drop of 'emails'--look for story URL in Content-Base header for Thunderbird RSS 'emails'.
- Apply 'email' proc flag when processing dragged .eml files--which are emails.
- Update translations.

[JimmXinu]

Originally Posted by ownedbycats

... (RSS stuff) ...

Okay, this amuses me.

The RSS "email" doesn't actually contain the story URL in the email body.

But i contains it in a header field, Content-Base:, which apparently is obsolete? At least in emails?

Anyway, I added it. Merry Christmas.

[tricklem]

Originally Posted by steltek

Go in to configure FFF. Go to the Personal.ini tab, then hit the "Edit Personal.ini" button. When Personal.ini opens up, look for the [defaults] section right near the top. Right below [defaults], enter a new line beginning with browser_cache_path: followed by your cache path (no spaces between the : and your browser cache path).

Then, go to the [overrides] section at the bottom of Personal.ini. Paste the following text below [overrides]:

[www.fanfiction.net]

##the following line tells FFF to use the browser cache when set to true

use_browser_cache:true

##the following line limits FFF to use cache entries no older than the
##number of hours shown (in this case, cached items no 4 hours or less old).

browser_cache_age_limit:4.0

Then, save the changes to Personal.ini using the OK button. Once you have done this, you should be able to browse a fic in Firefox, then run FFF to collect the pages out of the browser cache.

Did something wrong.

I am getting this error: cloudscraper reports: "Cloudflare IUAM possibility malformed, issue extracing delay value."

This is what I have entered under personal.ini

Under defaults:

Spoiler:

## browser_cache_path:C:\Users\myname\AppData\Local\M ozilla\Firefox\Profiles\466fs7zq.default-release\cache2

Then under overrides

[overrides]
## default varies by site. Set true here to force all sites to
## collect series.
#collect_series: true

##[www.fanfiction.net]
##use_browser_cache:true
##browser_cache_age_limit:4.0
##use_browser_cache_only:true

Can anyone tell me what I did wrong?

[ownedbycats]

Originally Posted by JimmXinu

Okay, this amuses me.

The RSS "email" doesn't actually contain the story URL in the email body.

But i contains it in a header field, Content-Base:, which apparently is obsolete? At least in emails?

Anyway, I added it. Merry Christmas.

Thank you!

Content-Base was used officially in earlier versions of the HTTP and MHTML standards, but deprecated in both. But it's still used as a non-standard header in some contexts such as emails.

One of my favourite things is RFC 2324.

[ownedbycats]

Originally Posted by tricklem

Did something wrong.

I am getting this error: cloudscraper reports: "Cloudflare IUAM possibility malformed, issue extracing delay value."

This is what I have entered under personal.ini

Under defaults:

Spoiler:

## browser_cache_path:C:\Users\myname\AppData\Local\M ozilla\Firefox\Profiles\466fs7zq.default-release\cache2

Then under overrides

[overrides]
## default varies by site. Set true here to force all sites to
## collect series.
#collect_series: true

##[www.fanfiction.net]
##use_browser_cache:true
##browser_cache_age_limit:4.0
##use_browser_cache_only:true

Can anyone tell me what I did wrong?

You left them commented out. Remove the ##s, except the two lines just above collect_series.