[GUI Plugin] FanFicFare

[estherflails]

Hello again!
It's been a little bit, but I wanted to thank you for all your help regarding merengo.hu. I finally had some time to figure out my personal.ini and download most of the fics I wanted.

I'm sorry to bother you about this again, but another issue popped up and I have no idea what might be causing it. Some stories are still getting errors when I try to download them. Here are some examples:

'NoneType' object has no attribute 'find_all' https://merengo.hu/viewstory.php?sid=10050
'NoneType' object has no attribute 'find_all' https://merengo.hu/viewstory.php?sid=80621
'NoneType' object has no attribute 'parent' https://merengo.hu/viewstory.php?sid=140935

'find_all' happens more often, 'parent' is relatively rare.

Comparison stories I could downloaded without problems:
https://merengo.hu/viewstory.php?sid=52296
https://merengo.hu/viewstory.php?sid=80371
https://merengo.hu/viewstory.php?sid=66968


Also, the titlepage of downloaded stories contains an extra section called 'Story Notes', which are usually the author notes for the first chapter.
Here's one where there are no author notes, but there are end notes, and those still end up on the title page instead of at the end of the chapter: https://merengo.hu/viewstory.php?sid=147485
This section also lacks any formatting like line breaks or bold text.
I tried to find a story that had author notes and end notes as well, to see what happens then, but had no luck.

I found the reason for this is this section in base_efiction_adapter:

	## Retrieving the story notes
        sn = soup.find('div', {'class':'noteinfo'})
        if sn:
            self.story.setMetadata('storynotes', stripHTML(sn))

I saw that other efiction sites have a section like this instead of story and endnotes.
By changing it to this, I managed to make it so only "Story Notes" labelled ones show up, like on this site: http://www.libraryofmoria.com/a/viewstory.php?sid=4023

        ## Retrieving the story notes
        sn = soup.find('div', {'class':'noteinfo'})
        if sn:
            if soup.find('span', {'class':'label'},text='Story Notes:'):
                self.story.setMetadata('storynotes', stripHTML(sn))

But I couldn't figure out how to add the "Author's Notes" and "End Notes" labelled parts into the correct places.
If this would be too complicated to fix, don't worry about it, it's not a huge deal.

Thank you!

[JimmXinu]

New Test Version Posted

2022-02-20
- base_efiction - narrow chapter search more--for merengo.hu.
- Stop looking for FFDL settings--it's been 7 years.

Originally Posted by estherflails

I'm sorry to bother you about this again, but another issue popped up and I have no idea what might be causing it. Some stories are still getting errors when I try to download them. Here are some examples:

'NoneType' object has no attribute 'find_all' https://merengo.hu/viewstory.php?sid=10050
'NoneType' object has no attribute 'find_all' https://merengo.hu/viewstory.php?sid=80621
'NoneType' object has no attribute 'parent' https://merengo.hu/viewstory.php?sid=140935

These all have similar problems: they were all finding 'chapter links' that weren't chapter links.

Strangely, they all three were finding 'extra' chapters in different ways. They range from not skipping the 'Back to index' links in Hungarian to author including HTML that base_efiction falsely flagged as chapter links.

I've made changes to account for these. Hopefully they won't break anything else...

Originally Posted by estherflails

Also, the titlepage of downloaded stories contains an extra section called 'Story Notes', which are usually the author notes for the first chapter.
Here's one where there are no author notes, but there are end notes, and those still end up on the title page instead of at the end of the chapter: https://merengo.hu/viewstory.php?sid=147485

I hesitate to make changes to base_efiction for this without a large sample set of stories with notes on different sites--base_efiction is used for 20+ different sites.

You can remove the bad story notes for yourself by putting this in your personal.ini:

[base_efiction]
add_to_extra_titlepage_entries:

Note that it does have to be [base_efiction] to override the add_to_ setting in defaults.ini.

[ownedbycats]

Dumb question: Is it possible to do a replace_metadata on an AO3 series based on URL? I have a series=> replacement that breaks every time the author decides to change the series name, lol.

[JimmXinu]

Should be, using seriesUrl.

[Ser4nb2LUY6e]

when using "update existing fanfiction books" function with "background metadata" enabled, any chance to get an additional "create one job per website" option?

[Ser4nb2LUY6e]

fastnovel.net has started changing the URLs every few days for some reason.

e.g. over the last week I've seen:

https://fastnovel.net/cultivation-chat-group6-29/
https://fastnovel.net/cultivation-chat-group7-29/
https://fastnovel.net/cultivation-chat-group8-29/

There is an automatic redirect in place from the old to the new URL, and FanFicFare is following that. So far so good.

But, while it updates the url in the metadata, it doesn't update all the url's in the chapter files. So what happens is that I run an "update epub" on any of the fastnovel books, and most of the time it will download one or two new chapters... and then suddenly it takes 4h because it's download 2500something chapters again from scratch instead of just updating their URL and loading the new ones.

If I go and edit the epub to update the chapter url stored in the .xhtml files by hand, then the update works and it's just downloading the new chapters.

Any way for FFF to recognize that a redirect on the main page should also be applied to all existing chapters instead of downloading them all from scratch again?

[JimmXinu]

Originally Posted by Ser4nb2LUY6e

when using "update existing fanfiction books" function with "background metadata" enabled, any chance to get an additional "create one job per website" option?

What's the benefit?

The BG job already launches a separate process for each site--limited by the number of CPUs Calibre reports, which Calibre also does to BG jobs.

The downsides are additional complexity and getting more than one 'Proceed to update library' dialog as each BG job ends. Which I'm sure will confuse some users, even if it's an option they have to turn on.

[Ser4nb2LUY6e]

Originally Posted by JimmXinu

What's the benefit?

The BG job already launches a separate process for each site--limited by the number of CPUs Calibre reports, which Calibre also does to BG jobs.

The downsides are additional complexity and getting more than one 'Proceed to update library' dialog as each BG job ends. Which I'm sure will confuse some users, even if it's an option they have to turn on.

Well, if something goes wrong with one of the websites (that results in FFF spinning it's wheels forever instead of erroring out, like with the fastnovel thing I posted), it doesn't mess up the download from the other sites.

Also, I have a 24-core (48-thread) ThreadRipper system. Based on the messages in the FlareSolverr window, I can say with absolutely certainty that on my system it never downloads anything in parallel.

(Also that, if I cancel the job because of that fastnovel thing, the job is shown as canceled after a few seconds, while in the background it will happily continue to download 2000+ chapters if I don't kill FlareSolverr.)

If I start separate jobs for the different websites by hand, it always finishes much faster than if I try to just start a single job for all the books I want to update.

Also, showing multiple 'Proceed to update library' messages is a feature not a bug. That way, once the first site finishes, I can start reading the new chapters while the other downloads are still running instead of waiting another 20 minutes or so for all of them to finish.

[JimmXinu]

New Test Version Posted

2022-02-22
- adapter_fastnovelnet: 'Normalize' chapter URLs to current storyId URL--site is changing it frequently

Originally Posted by Ser4nb2LUY6e

fastnovel.net has started changing the URLs every few days for some reason.
...
Any way for FFF to recognize that a redirect on the main page should also be applied to all existing chapters instead of downloading them all from scratch again?

Yet Another Site that only works through FlareSolverr. Joy.

Looking at a few stories from the Latest Releases list, I do see some that have a number in the URL, after the title part, before the '-idnum'.

You can trim down the URLs (and chapter URLs) to have no title at all (but keep the '-') and they work. IE, https://fastnovel.net/-29/ and https://fastnovel.net/-29/chapter-25206.html

But that's ugly--and obvious to anyone watching the server logs.

adapter_fastnovelnet keeps the title part of the URL in storyId too, 'cultivation-chat-group6-29', not just 29.

FFF already has a hook to 'normalize' chapter URLs, previously developed for base_xenforo. Using it with adapter_fastnovelnet to 'normalize' the chapter URLs to have the current version of storyId is possible.

It still gives you the 'Change Story URL?' dialog when the story URL changes. Which, to be honest, is the truth.

[Ser4nb2LUY6e]

Originally Posted by JimmXinu

Yet Another Site that only works through FlareSolverr. Joy.

I know... It's annoying.

Originally Posted by JimmXinu

adapter_fastnovelnet keeps the title part of the URL in storyId too, 'cultivation-chat-group6-29', not just 29.

FFF already has a hook to 'normalize' chapter URLs, previously developed for base_xenforo. Using it with adapter_fastnovelnet to 'normalize' the chapter URLs to have the current version of storyId is possible.

It still gives you the 'Change Story URL?' dialog when the story URL changes. Which, to be honest, is the truth.

Ok, the test version just updated one of the affected stories for me without any manual fiddling with the URLs. Great!

Interestingly, it first downloaded:

https://fastnovel.net/cultivation-chat-group7-29/

followed by:

https://fastnovel.net/cultivation-ch...r-1113534.html
(the new chapter that I didn't have yet).

whereas if I open the website in the browser, it forwards me to:

https://fastnovel.net/cultivation-chat-group8-29/

So I'm *not* getting an 'Change Story URL?' dialog. And it will keep downloading future chapters using the "old" URL.

I looked at the chapter that was downloaded, and it seems fine... But I could envision the site getting snarky in the future and sending out some trash when using the "old" url... I guess we'll have to cross that bridge when it's burning.

[JimmXinu]

New Test Version Posted

2022-02-22
- adapter_fastnovelnet: Update to redirected Story URL.
- adapter_fastnovelnet: 'Normalize' chapter URLs to current storyId URL--site is changing it frequently

Originally Posted by Ser4nb2LUY6e

Well, if something goes wrong with one of the websites (that results in FFF spinning it's wheels forever instead of erroring out, like with the fastnovel thing I posted), it doesn't mess up the download from the other sites.

The situation you bumped into, while admittedly annoying, is a corner case that shouldn't be all that common.

Originally Posted by Ser4nb2LUY6e

Also, I have a 24-core (48-thread) ThreadRipper system. Based on the messages in the FlareSolverr window, I can say with absolutely certainty that on my system it never downloads anything in parallel.

How many CPUs is your Calibre configured to use? It's a setting under Preferences -> Misc

I will investigate if FFF's parallel processing is still working as intended. It uses Calibre's Server class and I haven't looked at it in any detail in a while.

Originally Posted by Ser4nb2LUY6e

Also, showing multiple 'Proceed to update library' messages is a feature not a bug. That way, once the first site finishes, I can start reading the new chapters while the other downloads are still running instead of waiting another 20 minutes or so for all of them to finish.

Since you can already launch separate updates by site if you really want to, I'm not very inclined to add it as a new feature.

I honestly think we'd have more people asking questions after turning it on without understanding than we'd have people it would benefit.

Originally Posted by Ser4nb2LUY6e

So I'm *not* getting an 'Change Story URL?' dialog. And it will keep downloading future chapters using the "old" URL.

I looked at the chapter that was downloaded, and it seems fine... But I could envision the site getting snarky in the future and sending out some trash when using the "old" url... I guess we'll have to cross that bridge when it's burning.

That's the difference between selecting the book(s) and hitting update vs updating from URL. I assume most updates are from update notification emails with URLs, so that's what I tested.

I've addressed the redirect story URL change now while I still remember about it.

[JimmXinu]

Originally Posted by JimmXinu

I will investigate if FFF's parallel processing is still working as intended. It uses Calibre's Server class and I haven't looked at it in any detail in a while.

FYI, I'm not seeing anything I didn't expect.

I can see processes forking off and record the PIDs and when sorting the overall log data by time, I can see parallel processing happen. The output from Calibre is ordered by process, which is more useful 99% of the time.

Changing Calibre's "Max. simultaneous conversion/news download/plugin jobs:" setting changes the number of processes FFF's BG job will run at the same time

FFF ignores Calibre's "Limit the max. simultaneous jobs to the available CPU cores" setting. I don't know off hand if Calibre's Server class honors it under the covers.

[bugstomper]

I'm reading a book on RoyalRoad that has links to other chapters in the book. It's a choose your own path type adventure. Is there anything in FanFicfare that would support getting those links converted to the proper internal links in the resulting epub? If it is not already a feature, do you think that existing functions for matching a URL to a story URL pattern and canonicalizing a URL and whatever is done to produce a ToC could be the basis for coding an option to convert such links if I wanted to pursue coding this feature myself?

[JimmXinu]

FFF has code to normalize and internalize relative story links to chapters, the normalize_text_links and internalize_text_links options.

They are on by default, but we don't have a dedicated normalize_chapterurl() in adapter_royalroadcom, so if the author's URLs don't exactly match the chapter URLs, it won't work. I would guess there's a good chance adding an appropriate normalize_chapterurl() to adapter_royalroadcom may be necessary.

I don't read anything from RoyalRoad, so I don't know anything off hand about their linking. I'd need to see the story.

fix_relative_text_links is related, but also kind of the opposite--it changes relative links to absolute links so they work when not on the same site. It's done before normalize_text_links & internalize_text_links (so matching chapter links can still be inernalized) and is also on by default. I'm honestly not sure what it would do to #anchor only links within a page.

[JimmXinu]

New Test Version Posted

2022-02-24
- Change base_xenforoforum reveal_invisible_text feature to also add class=invisible_text. Closes #812