[GUI Plugin] FanFicFare

[JimmXinu]

New Site? readnovelfull.com

Originally Posted by junni

I've been fiddling around in python and got a rough understanding of it, so i tried to make an adapter for the readnovelfull.com website, since it is similar to novelfull.com. I got it working, but i am sure the code could be way pretty. Anyways, if anyone is interested on it, here it goes.

For a first attempt, this is remarkably good.

In fact, the only obvious flaw I found was with status processing; it didn't work with 'Ongoing' stories and FFF standardizes status to Completed and In-Progress. (Some sites do also have additional states.)

I changed it to:

        if soup.find("a", href=re.compile("/completed-novel"), class_="text-primary"):
            self.story.setMetadata("status", "Completed")
        else:
            self.story.setMetadata("status", "In-Progress")

However, I hesitate to add this site, because (like novelfull.com), it doesn't work except through flaresolverr. And as ffnet has shown, just because that works today, doesn't mean it will work tomorrow. I'm getting very tired of wrestling with cloudflare and even more tired of dealing with users about it.

That said, attached is a plugin test version containing junni's code (plus status change). If there are other users who would find this useful, ever with the flaresolverr limitation, I'll add it to the official repo.

Originally Posted by junni

P.S. i don't know how to use github, so i'm sorry if i can't post it there, or if i'm permitted to post here.

If you want to contribute regularly (which I do encourage), you should eventually take the time to learn to use github. But you wouldn't be the only contributor who didn't.

An attached zip file instead of a inline would be better, but I don't think MR allows that until you have 10 posts or something like that. At least you used [code] tags.

[junni]

Originally Posted by JimmXinu

New Site? readnovelfull.com



For a first attempt, this is remarkably good.

In fact, the only obvious flaw I found was with status processing; it didn't work with 'Ongoing' stories and FFF standardizes status to Completed and In-Progress. (Some sites do also have additional states.)

I changed it to:

        if soup.find("a", href=re.compile("/completed-novel"), class_="text-primary"):
            self.story.setMetadata("status", "Completed")
        else:
            self.story.setMetadata("status", "In-Progress")

Hi JimmXinu, with further testing i noticed that there was a bug in this code, because the website changes the HREF tag based on completed or ongoing novel . so i made a slight change which made it work. I should have tested further , but i was just too excited to post my accomplishment. .

try:
            status = soup.find("a", href=re.compile("/completed-novel"), class_="text-primary").text
        except:
            status = soup.find("a", href=re.compile("/ongoing-novel"), class_="text-primary").text		
		
        if status == "Completed":
            self.story.setMetadata("status", "Completed")
        else:
            self.story.setMetadata("status", "In-Progress")

[JimmXinu]

Originally Posted by junni

Hi JimmXinu, with further testing i noticed that there was a bug in this code, because the website changes the HREF tag based on completed or ongoing novel . so i made a slight change which made it work. I should have tested further , but i was just too excited to post my accomplishment. .

try:
            status = soup.find("a", href=re.compile("/completed-novel"), class_="text-primary").text
        except:
            status = soup.find("a", href=re.compile("/ongoing-novel"), class_="text-primary").text		
		
        if status == "Completed":
            self.story.setMetadata("status", "Completed")
        else:
            self.story.setMetadata("status", "In-Progress")

Isn't that the same issue I identified and fixed? Effectively, the results are the same, but my version is a bit short and faster, I think.

[MonicaOP]

Oh my!!! Yeah, I guess not of this is going to happen!!! Oh wow, the fact that everything I know about the internet is self taught and that I have no idea how to program, I'm so, so, so lacking in this world!!

Thank you so much for the ideas, but to show my ignorance once again, I have no idea how to add it to the github of FFF"" The site I want is https://www.novelhall.com/ 'cause it has a lot of completed stories that aren't in other sites, but I really don't know how to ask.

I'm really really sorry if I sound ignorant is not that I'm lazy or anything is just that I never really had the time to learn to do this things 'cause I never needed them before.

Thanks again, Be well, Monica

Originally Posted by fanfictioneer

First of all, you need to be able to program in python - ownedbycats posted the link directly to the place where you can check how the adapters are coded (basically, each site has an "adapter" that makes FFF able to download from it).

If you don't know how to program in python, you can always create an issue in the github for FFF, specifying which site it is that you want added. Note that generally, people mostly create adapters for sites that they are interested in reading themselves, so your issue might languish, unfixed, for quite some time. Your best bet to get a site added is if it is very similar to an existing supporting site, such as if it is a xenforo-forum site.

If you want to learn how to do it yourself, your best bet (IMHO) is studying one of the existing (preferably simpler) adapters and making sure you understand how that one functions, then trying to write your own adapter for the site you are interested in. Once you've got it working for the stories you are interested in, try it out for a few other stories on that site, too, then create a pull request for JimmXinu to add your adapter to FFF.

Good luck!

[Arkeus]

So, I thought that ffnet hadn't worked at all since april unless you count manually opening each chapter when you want to download them, but people have been talking like it worked again recently?

Did I miss something obvious?

[Ytierian]

Originally Posted by Arkeus

So, I thought that ffnet hadn't worked at all since april unless you count manually opening each chapter when you want to download them, but people have been talking like it worked again recently?

Did I miss something obvious?

It's been an on-again/off-again thing since April. They make a security change, Jimm finds a work around that functions for a bit...then they make a new security change.

Rinse-repeat. Latest was a couple weeks ago...FFF was working on FFN until then.

It's to the point that I think Jimm's getting tired of having to fix that issue every couple weeks - so he's tossed it to the community to see if someone else wants to bang their head against a wall for a while.

Don't look at me - my only programming languages are Basic, Cobol and RPG-II

[Arkeus]

Originally Posted by Ytierian

It's been an on-again/off-again thing since April. They make a security change, Jimm finds a work around that functions for a bit...then they make a new security change.

Rinse-repeat. Latest was a couple weeks ago...FFF was working on FFN until then.

It's to the point that I think Jimm's getting tired of having to fix that issue every couple weeks - so he's tossed it to the community to see if someone else wants to bang their head against a wall for a while.

Don't look at me - my only programming languages are Basic, Cobol and RPG-II

OK this is really weird, because I've been using fanficfare every day since april and haven't had a single ffnet update since then.

I... guess all the working updates must have been on 'beta' versions?

[ownedbycats]

Originally Posted by Arkeus

So, I thought that ffnet hadn't worked at all since april unless you count manually opening each chapter when you want to download them, but people have been talking like it worked again recently?

Did I miss something obvious?

I've been using the Flaresolverr proxy. Leaving it running in the background essentially let FFF work as usual, maybe a little slower. Unfortunately ffnet increased the CF level and broke this method.

[thenico]

Originally Posted by Arkeus

So, I thought that ffnet hadn't worked at all since april unless you count manually opening each chapter when you want to download them, but people have been talking like it worked again recently?

Did I miss something obvious?

Proxiying to a real browser worked before.
Last successful update was 2021-10-24 for me.

[Mandabar]

Originally Posted by ownedbycats

I've been using the Flaresolverr proxy. Leaving it running in the background essentially let FFF work as usual, maybe a little slower. Unfortunately ffnet increased the CF level and broke this method.

Yeah I started using that when it borked for me in early September, though it was working fine till then. Then 1.2.9 stopped working this last few days. The newer flaresolver 2.0/2.0.1 works and bypasses CF easily but unfortunately removed the download function which seems to be key to how FFF uses FS.

I see Jimm has an issue on their git for it so here is fingers crossed they re add that functionality back in.

Funnily enough, 1.2.9 still works.. occasionally. No idea why or how but got an update on a few of my fics luckily with it.

[ownedbycats]

Are you referring to the "Cloudflare and FanFiction.net doing its thing again" issue? Because that's been open since June, when ffnet broke cloudscraper.

[JimmXinu]

Originally Posted by Mandabar

The newer flaresolver 2.0/2.0.1 works and bypasses CF easily but unfortunately removed the download function which seems to be key to how FFF uses FS.

I didn't know flaresolver v2+ still worked, or worked again...

I have a version I haven't released that works with flaresolver v2+, but at the expense of no cover image downloads and making assumptions about encoding.

[ownedbycats]

...is it telling that I had a dream that fanfiction.net went poof and my dream-self went "oh well, at least Jim won't have to constantly fight with that site anymore"?

[bpothier]

Seeing an issue with SOL where "long" chapters are not fully loading - manually see it continue loading when you scroll to that point in browser.

e.g. SOL.../s/26150/the-gadgeteer - Chapter 3

FFF is instead only getting to the "There is more of this text..." and recording that. e.g.:

<div id="sr"><div class="c b"><i>There is more of this text...</i><br><a href="javascript:void(0)" onclick="ld1()" style="display:block;margin:15px auto auto auto;box-shadow:0 0 .3em #777;padding:5px;border-radius:.3em;font-weight:bold;font-family:sans-serif;text-align:center;max-width:15em">Click to Load text...</a></div></div>

The "ld1()", "fetchData()" and "fd()" JS looks to be defined in "SOL.../res/js/min.story.1.2.js"

function ld1(){var b=true;if(b){var a=story_id+5;fetchData("tl","gt",[story_id,pid,ci,a],sver);b=false}}

This appears to be a "recent" site update (last few days?) - guessing there is/was something similar previously, so hopefully just a quick fix...

For everyone else, you can check for the "There is more" text inside recent EPUBs with something like:

find . -mtime -3 -name \*.epub | while read EPUB ; do echo "=====> ${EPUB}"; unzip -c "${EPUB}" | egrep "fff_chapter_title|There is more of this text" | grep -B1 "There is more of this text"; done

This will list which chapters are "incomplete", you can then edit the EPUB, delete those chapters and re-download (once a fix is confirmed..)

[JimmXinu]

New Test Version Posted

2021-11-01
- adapter_royalroadcom: fix for ancient bug reading unixtime attr that's come back after years. #753

FYI: The posted test version does not support readnovelfull.com due to lack of response asking for it. I said before:

Originally Posted by JimmXinu

However, I hesitate to add this site, because (like novelfull.com), it doesn't work except through flaresolverr. And as ffnet has shown, just because that works today, doesn't mean it will work tomorrow. I'm getting very tired of wrestling with cloudflare and even more tired of dealing with users about it.

That said, attached is a plugin test version containing junni's code (plus status change). If there are other users who would find this useful, ever with the flaresolverr limitation, I'll add it to the official repo.

All of the above, plus current versions of flaresolverr can't download images.