[GUI Plugin] FanFicFare

[roguexuk]

Originally Posted by JimmXinu

New Test Version Attached
(Here, not in the usual place.)

2020-12-22

However, it does still work in older versions for other sites. As far as I know. This is more 'test' than usual.

UPDATE: I've written in more detail here, in the github issue.

Hi,

I just tried it on Calibre 4.23 (Win10) and it didn't work for me:

Status
Title Stranger Than Fiction (Avengers)
Author Neuropsych
Comment Detected a Cloudflare version 2 Captcha challenge, This feature is not available in the opensource (free) version.
URL https://www.fanfiction.net/s/13747370/1/
Error

[JimmXinu]

Originally Posted by JimmXinu

This isn't 100% and probably won't help with ffnet in older Calibre versions.

However, it does still work in older versions for other sites. As far as I know. This is more 'test' than usual.

UPDATE: I've written in more detail here, in the github issue.

From the github issue I linked to:

Cloudscraper in Calibre Plugin

I needed to bundle 6 additional modules into the plugin, and change the cloudscraper code in one place, but it works.

To my surprise, it even works in older Calibre versions. Or, at least, it runs. All the old versions I tested (4.23, 3.48, 2.85.1) give some variation of this error just importing cloudscraper:
DEPRECATION: The OpenSSL being used by this python install (OpenSSL 1.0.2s 28 May 2019) does not meet the minimum supported version (>= OpenSSL 1.1.1) in order to support TLS 1.3 required by Cloudflare, You may encounter an unexpected Captcha or cloudflare 1020 blocks.
...just with increasingly older versions of OpenSSL.

I didn't actually test the cloudscraper plugin version on versions older than 4.23 because it was already failing and giving:
CloudflareChallengeError: Detected a Cloudflare version 2 Captcha challenge, This feature is not available in the opensource (free) version.

It is unfortunate, but entirely possible, that ffnet support may not be possible in older versions of Calibre from now on.

[roguexuk]

Hi Jimm

I updated Calibre to 5.7.2 with your test plug-in & it worked!!
Thank you so much :-)

[ownedbycats]

I installed the test version and ran tests on a few random fics to see if I came across any errors. I got a nonetype error once, but running it on that fic a second time gave me the usual "already has 6 chapters..." message.

If I come across any fics that consistently produce errors, I'll post the links. But tentatively I'd say it's working.

[JimmXinu]

New Test-Test Version
Instead of the usual place:
https://www.mobileread.com/forums/sh...88#post4072388

2020-12-22
- Adding cloudscraper and dependencies
- Updates for many included dependencies
- Need to semi-manually raise HTTPError for error codes with cloudscraper/requests.
- Temp: dump ffnet page to debug when metadata parsing fails.
- Clean up cloudscraper import
- Use requests.util.extract_zipped_paths to get browsers.json instead of kludge.

[ownedbycats]

Of course, now that you've gone to all this effort to fix it, what are the chances that FictionPress will reduce the CloudFlare protections in a few days?

[JimmXinu]

Yet Another New Test-Test Version
Instead of the usual place:
https://www.mobileread.com/forums/sh...88#post4072388

2020-12-22
- Adding cloudscraper and dependencies
- Updates for many included dependencies
- Need to semi-manually raise HTTPError for error codes with cloudscraper/requests.
- Temp: dump ffnet page to debug when metadata parsing fails.
- Clean up cloudscraper import
- Use requests.util.extract_zipped_paths to get browsers.json instead of kludge.
- Don't use mobile User-Agents with cloudscraper--adapter_fanfictionnet doesn't handle mobile pages.

I figured out why it was giving 'other' HTML sometimes--randomly selected mobile browser User-Agents.

[sallyhop1]

I have gotten this error for the last few days. Title:
Status Title Author Comment URL Error Unknown Unknown HTTP Error 403:
Forbidden https://www.fanfiction.net/s/13773051/1/

Status in calibre:
Invalid

Bug description:
I have gotten this error for the last few days. I've done the latest
update, but it is still happening on more than one story. I am using
version 5.7.2 64 bit; Windows 8.1. I was trying to download as a
Kindle file.

Thanks!

[ownedbycats]

Originally Posted by sallyhop1

I have gotten this error for the last few days. Title:
Status Title Author Comment URL Error Unknown Unknown HTTP Error 403:
Forbidden https://www.fanfiction.net/s/13773051/1/

Status in calibre:
Invalid

Bug description:
I have gotten this error for the last few days. I've done the latest
update, but it is still happening on more than one story. I am using
version 5.7.2 64 bit; Windows 8.1. I was trying to download as a
Kindle file.

Thanks!

See the last few pages: https://www.mobileread.com/forums/sh...d.php?t=259221

tl;dr: FFnet was getting DDoSed and added Cloudflare. Cloudflare detects FFF as a bot and blocks it. JimmXinu made a test version of FFF that doesn't get blocked.

[bpothier]

Originally Posted by JimmXinu

I just went to checkout Storiesonline and it gave me the Story not found in Author's list--Set Access Level to Full Access and change Listings Theme back to Classic error.

Looking at an author list, I was not seeing 'Classic' list theme.

My user account on Storiesonline already showed Classic; but when I changed it to something else and then back again, Classic was restored and FFF works again.

Saw same issue, though I did not see interface change to "classic" SoL style (Num/Title/Description/Size/etc.. table/columns view) after changing it back to classic. However, going to another author page did show the change, so guessing it was a browser cache issue - but FFF/Calibre would have used correct view. So, *YES!* change view to something else, save then change it back to "classic" works for me as well! FYI: The view i changed by clicking the "Settings" link in top right that only shows up on certain "listings" pages (e.g. Stories accessed today, Top xxxxx list, etc.. or in profile page click the "Change" link next to Listings section)

[ownedbycats]

edit: delete

[bpothier]

StoriesOnline "dupe" story ID/URL re-use issue...
Not sure how/if this gets addressed for other sites, but SoL occasionally "archives"(deletes) stories (for various reasons). Once a story is fully archived/deleted, its story ID/URL (e.g. /12345) is eligible for re-use. FFF only looks at the story/URL to determine if a story is "new" or "updated". Also looking at story title or author likely would not help as sometimes those are updated for various reasons... When a story ID is re-used, it results in errors like "Already contains 1 chapters" or an error about current containing more chapters than online version. Doing a Calibre search by the SoL ID will show a completely different story than the one it is trying to download - ID re-use.
My "fix" is to manually change the "old" story URL in Calibre to include the "publication" date for uniqueness, e.g. something like "/12345-2020-12-01" and just download the "new" one again. I also create a "SOL:" identifier in Calibre with that same format. Initially I just added "/12345-old" but then I started getting some stories that were re-used again! Unfortunately, this is an entirely manual process... Not sure if FFF could create this identifier automatically and if it could, could it be used to actually differentiate between "/12345", "/12345-2020-12-20", and "/12345-2017-11-14"? As these could potentially all have existed as 3 different stories from 3 different authors....
As a test, you should be able to simulate the issue by downloading some story with FFF, changing its URL to match another story and then try to download that 2nd story. FFF should reject it as already existing (either with same/more chapters). I haven't looked too closely to see what happens if old story had 1 chapter and new one has more? will it "update" the story and add missing chapters to end of existing story?... The other errors are more noticeable since it complains that it did not/would not download/update for equal/more local chapter reasons so I update the URL and retry..
(I can already imagine JimmXinu's dislike for SoL flaring up, so maybe someone else has an idea/fix? )

[ownedbycats]

add_to_replace_metadata:
## Changes breaks to paragraphs in comments for better formatting
 description=><br/>=></p><p>
 description=><p>=><p align="justify">

The second line doesn't work. Did I break the syntax?

[darth1]

Thanks for your hardwork jim, it works with my 3.14 for the ffnet stuff as I rarely use storyonline so i don't know about it

[tricklem]

Can someone advise me on how to install the test version into Calibre? I am running the newest version of Calibre/