[GUI Plugin] FanFicFare

[tricklem]

I only tried to update 1 book and got the clouldflare error and I have the newest version of calibre & fff. I updated yesterday no problem, today problem. What the heck is ff.net doing now?!

Detected a Cloudflare version 2 challenge, This feature is not available in the opensource (free) version.

[ownedbycats]

Rebooting my modem didn't work, when it had before when the original Cloudflare issues occured.

People are reporting other Cloudflare errors as well, on browser. I'm thinking they might've bumped Cloudflare up a level with that latest migration.

[JimmXinu]

Looks like this is happening to basically everybody. Further discussion in #622

My last response there was:

For reference: #614 was the last issue about ffnet and cloudflare.

Here are the avenues I have explored, none have worked so far :

• Turning cloudscraper off - Back to 503 error.
• Use different JS engine in cloudscraper - Doesn't get that far.
• Use captcha service in cloudscraper - Doesn't get that far.
• Copy cookies (& User-Agent) from browser - Same exception. Also, one of the cookies is set to expire after only 30 minutes.
• Comment out check that raises the exception - Fails to process farther on--cloudscraper really doesn't know how to handle it.

The exception suggests the existence of a pay or closed source version of cloudscraper, but I'm not finding it. Nor do I know if we could use it.

It is possible that the restrictions will relax over time. It is also possible that they will not. Other downloaders appear to be having the same problems.

[milford833]

Do not know if this helps to resolve it or not, I used fanfiction downloader, (the latest version) just a hour ago and it would download them, but it would error out when switching to PDF, but it downloaded the HTMLs, which I now have all of the ones I wanted in HTML.

[ownedbycats]

Originally Posted by milford833

Do not know if this helps to resolve it or not, I used fanfiction downloader, (the latest version) just a hour ago and it would download them, but it would error out when switching to PDF, but it downloaded the HTMLs, which I now have all of the ones I wanted in HTML.

We're discussing that in the GitHub thread. They're closed-source, but it seems they use a different method than FFF for scraping the content from the website.

[ownedbycats]

My best recommendation:

Convince as many authors as possible to crosspost to AO3.

If you can't find them on AO3 (search fanfic titles, in case they use a different username) - send them a PM on FFNet, offering them an invite. Mention the benefits of AO3: better tagging, mostly, but also being able to moderate comments and group fanfics together in series. And that FFnet is kind of falling apart. Also appeal to every author's greatest desire - new readers!

And then offer them an AO3 invite code if they want one. (Of course, this means you'll also need an account...feel free to bug me for one.)

And absolutely, under no circumstances harrass them about it. If they say no, leave it at that. If they don't respond, don't send them multiple PMs (note that ffnet also broke email notifications of PMs, so it may take a while*). Be nice about it.

* EDIT: PM notifications are working again, it seems.

[Mandabar]

Originally Posted by milford833

Do not know if this helps to resolve it or not, I used fanfiction downloader, (the latest version) just a hour ago and it would download them, but it would error out when switching to PDF, but it downloaded the HTMLs, which I now have all of the ones I wanted in HTML.

I wonder how amenable they would be to open-sourcing out that component of their software. I mean we are all fanfic lover together. Whether it can work as a calibre plugin is another story.

[ownedbycats]

The method that FFDL (the standalone one, not the old FFF) seems to use is rather...expensive.

Basically, they open an invisible (headless) browser and use that for Cloudflare bypass/scraping.

It works okay for FFDL because its GUI makes it clear that it's really only intended to download one fic at a time. But when you're checking/updating multiple fics at once, as FFF is intended for? it might be a bit more painful.

[JimmXinu]

I would point out that this is ownedbycats speaking for themself. (Large type size does not confer authority.)

As the maintainer of FanFicFare, I would prefer if the FFF community did not get a reputation for telling authors what do to.

Originally Posted by ownedbycats

My best recommendation:

Convince as many authors as possible to crosspost to AO3.
...
And absolutely, under no circumstances harrass them about it. If they say no, leave it at that.

I would further say that bugging anyone until they respond to say no is also harassment.

[ownedbycats]

I changed it to bold and added some stuff to the last paragraph to make it less confusion.

[ownedbycats]

Question: are regexes used in replace_metadatas case-sensitive or can be made case-sensitive?

Another question: Also, while changing over one of my fics to AO3 (I just swapped the url: identifier over), it overwrote the ffnet cover I had with an autogenned one in spite of this:

[https://archiveofourown.org/works/1036707/]
never_make_cover:true

My other never_make_covers work. It's at the the bottom of personal.ini. Did I make a mistake?

[willowtree3145]

FFNET is barfing again. Sorry to be a pest, but it looks like the Calibre plugin isn't doing so hot with fanfiction.net again. I tried updating a book and it gave me this error message:

Error: Detected a Cloudflare version 2 challenge, This feature is not available in the opensource (free) version.

Deleting and trying to re-download the book produces the same error message.

I checked the site and it seems to be working, as does my link. As far as I know, FFNET doesn't have a paid version (and the legality of that would be questionable, but that's beside the point), so, uh, help us please, supreme overlord?

Thanks,

Willowtree

[ownedbycats]

The 'free version' is referring to Cloudscraper. FFNet bumped up Cloudflare a level or two (not sure if permanent or not) and the developer of Cloudscraper puts that level behind a subscription model. Jim's looking into alternatives.

If you at least want access to the fic and it's not crossposted to AO3 (search both the title and author), the standalone FanFictionDownloader (no relation to the old version of FFF) uses a different method of scraping and is reported to be working.

[kim67255]

I just tried twilighted and updated 12 stories. It seems to be working now.

[ClairePMR]

Still getting the Cloudflare error as of 2:15 p.m. ET, so it's not working for everyone. Bummer!