[GUI Plugin] FanFicFare

[Mandabar]

Originally Posted by JimmXinu

I didn't know flaresolver v2+ still worked, or worked again...

I have a version I haven't released that works with flaresolver v2+, but at the expense of no cover image downloads and making assumptions about encoding.

Ah, well watching the console run, it seems to quickly respond to requests and say it solved the captcha or whatevers. I don't get results from FFF because, well it's not designed to work with the new FS nor it's lack of the download option.

If it is actually working or not? I can't say conclusively.
------
So I tried the test version for the fix to the new/old Royalroad bug. Working again so that's good!

[Garasou]

Originally Posted by bpothier

Seeing an issue with SOL where "long" chapters are not fully loading - manually see it continue loading when you scroll to that point in browser.

e.g. SOL.../s/26150/the-gadgeteer - Chapter 3

FFF is instead only getting to the "There is more of this text..." and recording that. e.g.:

<div id="sr"><div class="c b"><i>There is more of this text...</i><br><a href="javascript:void(0)" onclick="ld1()" style="display:block;margin:15px auto auto auto;box-shadow:0 0 .3em #777;padding:5px;border-radius:.3em;font-weight:bold;font-family:sans-serif;text-align:center;max-width:15em">Click to Load text...</a></div></div>

The "ld1()", "fetchData()" and "fd()" JS looks to be defined in "SOL.../res/js/min.story.1.2.js"

function ld1(){var b=true;if(b){var a=story_id+5;fetchData("tl","gt",[story_id,pid,ci,a],sver);b=false}}

This appears to be a "recent" site update (last few days?) - guessing there is/was something similar previously, so hopefully just a quick fix...

For everyone else, you can check for the "There is more" text inside recent EPUBs with something like:

find . -mtime -3 -name \*.epub | while read EPUB ; do echo "=====> ${EPUB}"; unzip -c "${EPUB}" | egrep "fff_chapter_title|There is more of this text" | grep -B1 "There is more of this text"; done

This will list which chapters are "incomplete", you can then edit the EPUB, delete those chapters and re-download (once a fix is confirmed..)

I'm honestly just going to have FFF force the full download of all the stories updated after date X when/if it's updated to get around this change to the site as having to manually edit each epub is way more work then just having FFF chug along in the background downloading everything.

[ownedbycats]

For what it matters, you can re-force the download of only specific chapters by opening it in ebook-editor and removing the files for them and then running a regular update. I do it sometimes myself.

I don't believe it'll actually save any time - FFF downloads all the chapters anyways IIRC - but it's helpful if you don't want to overwrite the entire thing.

[Da Man]

Originally Posted by JimmXinu

New Test Version Posted

2021-11-01
FYI: The posted test version does not support readnovelfull.com due to lack of response asking for it. I said before:

Just tried that site, I'm interested.

[davidfor]

Originally Posted by bpothier

Seeing an issue with SOL where "long" chapters are not fully loading - manually see it continue loading when you scroll to that point in browser.

e.g. SOL.../s/26150/the-gadgeteer - Chapter 3

FFF is instead only getting to the "There is more of this text..." and recording that. e.g.:

<div id="sr"><div class="c b"><i>There is more of this text...</i><br><a href="javascript:void(0)" onclick="ld1()" style="display:block;margin:15px auto auto auto;box-shadow:0 0 .3em #777;padding:5px;border-radius:.3em;font-weight:bold;font-family:sans-serif;text-align:center;max-width:15em">Click to Load text...</a></div></div>

The "ld1()", "fetchData()" and "fd()" JS looks to be defined in "SOL.../res/js/min.story.1.2.js"

function ld1(){var b=true;if(b){var a=story_id+5;fetchData("tl","gt",[story_id,pid,ci,a],sver);b=false}}

This appears to be a "recent" site update (last few days?) - guessing there is/was something similar previously, so hopefully just a quick fix...

For everyone else, you can check for the "There is more" text inside recent EPUBs with something like:

find . -mtime -3 -name \*.epub | while read EPUB ; do echo "=====> ${EPUB}"; unzip -c "${EPUB}" | egrep "fff_chapter_title|There is more of this text" | grep -B1 "There is more of this text"; done

This will list which chapters are "incomplete", you can then edit the EPUB, delete those chapters and re-download (once a fix is confirmed..)

That sounds like you not logged in. There was a recent change to the login page. Try the test version of FFF in the second post. It seems to work.

[Da Man]

Just tried readnovelfull.com with the old test version of FFF including that site and got
HTTP Error in FFF '403 Client Error: Forbidden for url: https://readnovelfull.com/age-of-cos....html'(403)

Reading online works fine.

[Ser4nb2LUY6e]

wuxiaworld.site is giving me:

'NoneType' object has no attribute 'i'

errors for all stories I try to update.

[JimmXinu]

Originally Posted by bpothier

Seeing an issue with SOL where "long" chapters are not fully loading - manually see it continue loading when you scroll to that point in browser.

Just saw this; MR has been very poor about notifying me of new posts lately.

If you click the 'Why' link right after 'Click to Load text...' it says:

New site behaviour: Click to Read more
Lazeez Jiddan (Webmaster)
11/2/2021, 8:15:44 AM

Due to abuse by bots downloading thousands of stories in very short time using hundreds, if not thousands, of free accounts, the site has come under a lot of strain and it's costing money in bandwidth and servers to keep the site running smoothly.

So in attempt to discourage some of those bots, we implemented a new measure. For free accounts (that's what bots use), a normal click on a story doesn't load the full text. It loads a portion and then at the end of that portion now you have to click two buttons to load the rest of the text.

So as of now, you can't right click on a link to save the complete text to your computer. Some browsers will save the whole text after you click the button to load it, and some will only save the chopped text. We can't control browser behaviour.

We are very well aware that it's annoying, and may deter some good folks from visiting the site. It's not something we planned to do, but we were forced to do it. As usual in life, a tiny minority of abusers make things worse for the good vast majority.

The 'Click to Load text...' link calls a javascript function, which POSTs to the server, which makes another link ('Click just one more time to Load text...') which calls another javascript function, which makes another POST to the server to get the remaining text.

The notice says that it only happens for free accounts, can we get a confirmation from a paid account holder that they don't see it in this chapter(https://storiesonline.net/s/26150:24...he-gadgeteer)?

Considering this site is original fiction and not fan fiction, I hesitate to attempt to work around this for free accounts.

[mrEd]

can we get a confirmation from a paid account holder that they don't see it in this chapter

Just bought a paid account, can confirm it's not showing.

[JimmXinu]

Multi-reply

Originally Posted by ownedbycats

I don't believe it'll actually save any time - FFF downloads all the chapters anyways IIRC - but it's helpful if you don't want to overwrite the entire thing.

For most sites, FFF does not download all chapters on update. And even when it does (mostly AO3), it does not replace existing chapters unless it has to.

This is considered a feature for users who sometimes change/correct chapter text/html.

Originally Posted by Da Man

Just tried readnovelfull.com with the old test version of FFF including that site and got
HTTP Error in FFF '403 Client Error: Forbidden for url: https://readnovelfull.com/age-of-cos....html'(403)

Did you do it through flaresolverr? That's the big reason I hesitate to add readnovelfull.com--having to explain over and over 'brand new site only works through flaresolverr--and FS v1.2.9, not latest.'

Originally Posted by Ser4nb2LUY6e

wuxiaworld.site is giving me:

'NoneType' object has no attribute 'i'

errors for all stories I try to update.

I have to put wuxiaworld.site through flaresolverr before I get to that error. (I'm noticing a trend here I don't much like.)

Back in Sept, wuxiaworld.site changed from having the chapter list in a separate POST req to putting in the story page directly. Now, they appear to have changed back, but with a different POST url without any request parameters.

However, it's not working when I try from FFF. Not sure if that's because of something subtle I'm missing or because flaresolverr won't accept a POST with no parameters.

If somebody else wants to look at it, great; but I don't have the energy or interest for this right now.

[Ser4nb2LUY6e]

Originally Posted by JimmXinu

The notice says that it only happens for free accounts, can we get a confirmation from a paid account holder that they don't see it in this chapter(https://storiesonline.net/s/26150:24...he-gadgeteer)?

Considering this site is original fiction and not fan fiction, I hesitate to attempt to work around this for free accounts.

It's not showing for paying users.

Paying users can also already download all the stories as nicely formatted epubs directly from the site without having to stress the server with retrieving it chapter by chapter.

[Ser4nb2LUY6e]

Originally Posted by JimmXinu

I have to put wuxiaworld.site through flaresolverr before I get to that error. (I'm noticing a trend here I don't much like.)

Back in Sept, wuxiaworld.site changed from having the chapter list in a separate POST req to putting in the story page directly. Now, they appear to have changed back, but with a different POST url without any request parameters.

However, it's not working when I try from FFF. Not sure if that's because of something subtle I'm missing or because flaresolverr won't accept a POST with no parameters.

If somebody else wants to look at it, great; but I don't have the energy or interest for this right now.

Fair enough, I can understand how this can get tiring.

Maybe there is one single thing you could still quickly test: just add some nonsense parameter to the POST.

[JimmXinu]

Originally Posted by Ser4nb2LUY6e

It's not showing for paying users.

Paying users can also already download all the stories as nicely formatted epubs directly from the site without having to stress the server with retrieving it chapter by chapter.

Originally Posted by mrEd

Just bought a paid account, can confirm it's not showing.

In that case, I'm more inclined to add something to the adapter that detects and errors when it sees the 'Click to Load text...' link instead of working around it. Opinions?

(FYI, we're talking about storiesonline.net--I'd already forgotten. Too many things going on. )

[Garasou]

Originally Posted by JimmXinu

In that case, I'm more inclined to add something to the adapter that detects and errors when it sees the 'Click to Load text...' link instead of working around it. Opinions?

(FYI, we're talking about storiesonline.net--I'd already forgotten. Too many things going on. )

Makes sense, at the time of my report the banner explaining the change wasn't up so I wasn't aware of the reasons for it. And since people are reporting that it goes away on a paid account I just will buy a sub to help the site while maintaining my use of FFF.

[readerboy7]

Edit:
I looked at the repo and seems issue was fixed by the time I posted this. I assume the fix will be in v4.7.0?

orig:

I'm getting the following error when downloading any fic from RoyalRoad, couldn't find discussion in last few pages.
NameError: name 'ts' is not defined
Traceback:

  File "calibre_plugins.fanficfare_plugin.jobs", line 260, in do_download_for_worker
  File "C:\Users\NAME\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\adapters\base_adapter.py", line 308, in getStoryMetadataOnly
  File "C:\Users\NAME\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\adapters\base_adapter.py", line 412, in doExtractChapterUrlsAndMetadata
  File "C:\Users\NAME\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\adapters\adapter_royalroadcom.py", line 161, in extractChapterUrlsAndMetadata
  File "C:\Users\NAME\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\adapters\adapter_royalroadcom.py", line 64, in make_date
NameError: name 'ts' is not defined

FanficFare v4.6.0
Calibre 5.31.1

Issue didn't seem to occur yesterday.