[GUI Plugin] FanFicFare

[JimmXinu]

New Test Version Posted

2021-12-11
- Add remove_class_chapter feature, true by default. For Calibre convert with Base eFiction.

This doesn't necessarily solve tricklem's problem--base efiction sites epubs which were converted epub to epub are likely broken.

But it should keep it from happening again (assuming the epub has been updated by FFF since this update) and should stop it from happening for Kindle users (again, to stories updated with this version and newer).

[tricklem]

Originally Posted by JimmXinu

I think the problem is an interaction between Calibre's Convert and the HTML from the site.

The epub you attached is output from Calibre Convert (epub->epub?)--not FFF.

Looking at your epub, the text is there, but the chapters have all been split into more than one file inside the epub. Just next page/page down from the chapter title and the text starts in a separate file.

A bit of testing shows that is what a Calibre epub -> epub conversion does to the FFF output from starslibrary.net.

Calibre's convert settings for 'Structure detection' is looking for tags with class="chapter". Which starslibrary.net (and all Base eFiction sites ) have marking the <div> tag containing the chapter text. By default (I assume--I don't remember changing that), Calibre is adding a 'pagebreak' there, apparently implemented as a break between files. Which it puts between the chapter title and the chapter text.

Trying to update such an epub gives me "FanFicFare doesn't recognize chapters in existing epub...". Which makes sense--it doesn't match FFF's file structure anymore.

So, the answer is, unfortunately, that any FFF Base eFiction site epub you did a calibre epub->epub convert on is likely broken as far as FFF update is concerned.

The only immediate fix is 'Don't do that.' Which isn't a great answer for people who convert to azw3 for Kindle (like me).

You have to go into Calibre's Conversion, Common options, under Structure detection and change the Chapter mark setting to 'None'. That will prevent new conversions from splitting the chapters.

I will add code to FFF to remove that class attribute for future downloads. But I need to think about it more first to decide if it's worth adding code to change existing downloaded chapters also.

I suspect this is a (somewhat) recent change to Calibre's default settings for Conversion? I feel like I would have run into it before working on base efiction sites, since I do use conversion...

I have made the change that you advised, just tested it out. It WORKED!

It must have defaulted to that setting when I did the update that wiped out my libraries.

You are awesome!

[Mandabar]

As a notice for those of you using flaresolver for fanfiction.net, you will need to update to newest release as of today for it to continue to work, version 2.1.0.
One of the lines on the changelog shows that they fixed the bypass for the new protections.
* Bypass new Cloudflare protection

I noticed it wasn't working Saturday with my weekly story that updates then. Luckily only had to wait till today for an updated version.

[ownedbycats]

Thanks for letting us know.

[JimmXinu]

Flaresolverr v2.1.0 worked just now when I tested it.

(Still doesn't work with images, though.)

[ownedbycats]

Question: Short of editing the database directly, is it possible to export FFF's settings from one library and import to another?

The text in the view library preferences dialog isn't editable (likely for the best; json is extremely easy to break) but I failed to see an import/export option anywhere.

[agumon1123]

Has anyone else noticed their reject URLs list not working on AO3 emails? it only happens on AO3 for me and I have discard URLs on reject list on but their not discarding them. It's been happening for about a week and I didn't change any settings between the day it was working fine and the day it stopped working. I think AO3 might have changed their emails someway.

[JimmXinu]

Originally Posted by ownedbycats

Question: Short of editing the database directly, is it possible to export FFF's settings from one library and import to another?

If you run calibre in debug mode, FFF will allow you to edit the settings JSON directly. With all the accompanying risk.

When you create a new library and switch to it, then go into FFF settings, FFF is supposed to copy it's settings from the previous library you just left. But I confess probably I haven't tested that in years.

Originally Posted by agumon1123

Has anyone else noticed their reject URLs list not working on AO3 emails? it only happens on AO3 for me and I have discard URLs on reject list on but their not discarding them. It's been happening for about a week and I didn't change any settings between the day it was working fine and the day it stopped working. I think AO3 might have changed their emails someway.

I may have seen something like that--I believe I was offered the reject yes/no dialog when 'Discard URLs on Reject List' checked. But I don't know what site it was for.

I don't have any AO3 update notice emails handy. I could use to see one, or a URL for a story that updates very frequently.

[tricklem]

Originally Posted by JimmXinu

I think the problem is an interaction between Calibre's Convert and the HTML from the site.

The epub you attached is output from Calibre Convert (epub->epub?)--not FFF.

Looking at your epub, the text is there, but the chapters have all been split into more than one file inside the epub. Just next page/page down from the chapter title and the text starts in a separate file.

A bit of testing shows that is what a Calibre epub -> epub conversion does to the FFF output from starslibrary.net.

Calibre's convert settings for 'Structure detection' is looking for tags with class="chapter". Which starslibrary.net (and all Base eFiction sites ) have marking the <div> tag containing the chapter text. By default (I assume--I don't remember changing that), Calibre is adding a 'pagebreak' there, apparently implemented as a break between files. Which it puts between the chapter title and the chapter text.

Trying to update such an epub gives me "FanFicFare doesn't recognize chapters in existing epub...". Which makes sense--it doesn't match FFF's file structure anymore.

So, the answer is, unfortunately, that any FFF Base eFiction site epub you did a calibre epub->epub convert on is likely broken as far as FFF update is concerned.

The only immediate fix is 'Don't do that.' Which isn't a great answer for people who convert to azw3 for Kindle (like me).

You have to go into Calibre's Conversion, Common options, under Structure detection and change the Chapter mark setting to 'None'. That will prevent new conversions from splitting the chapters.

I will add code to FFF to remove that class attribute for future downloads. But I need to think about it more first to decide if it's worth adding code to change existing downloaded chapters also.

I suspect this is a (somewhat) recent change to Calibre's default settings for Conversion? I feel like I would have run into it before working on base efiction sites, since I do use conversion...

Just in case anyone else is or does have this issue.....I spoke too soon because I was still having this issue with FF.net, so I went into Preferences and under Adding Books unchecked Automatically Convert Books to the Preferred Output Format under adding Actions and under Automatic Adding. It seems to have fixed the issue as I just tried with updating a story from FF.net.

[JimmXinu]

New test version posted

2021-12-14
- adapter_archiveofourownorg: AO3 notification emails now sending http: instead of https:

Originally Posted by JimmXinu

I don't have any AO3 update notice emails handy. I could use to see one, or a URL for a story that updates very frequently.

Got one. I do see the Reject dialog coming up because the chapter URL as returned from the email and isn't in the reject list on the first pass.

Ah, they've started sending http: instead of https: URLs. No idea why, but I have a fix. Assuming that's what you (agumon1123) are seeing, this should fix it.

Originally Posted by tricklem

Just in case anyone else is or does have this issue.....I spoke too soon because I was still having this issue with FF.net, so I went into Calibre Preferences and under Adding Books unchecked Automatically Convert Books to the Preferred Output Format under adding Actions and under Automatic Adding. It seems to have fixed the issue as I just tried with updating a story from FF.net.

Good point. I would add a word, for clarification.

[ownedbycats]

Originally Posted by JimmXinu

Ah, they've started sending http: instead of https: URLs. No idea why, but I have a fix.

That's... odd, because I was under the impression that AO3 redirected to https anyways.

[chrisric]

Hi,

Is there a way to make "use_browser_cache:true" global? I tried adding it for www.novelfull.com and got the error saying it's only for 2 specific sites.

If not, is there a way to point FFF to webtoepub generated epubs for updates? Yes, thats ugly but not sure what else to do except full manual processing.

I tried to get flareresovlerr going but its beyond me (just FYI, not looking for support), thats why I've gone this route.

Thanks.

-Chris

[JimmXinu]

Originally Posted by chrisric

Is there a way to make "use_browser_cache:true" global? I tried adding it for www.novelfull.com and got the error saying it's only for 2 specific sites.

If not, is there a way to point FFF to webtoepub generated epubs for updates? Yes, thats ugly but not sure what else to do except full manual processing.

I tried to get flareresovlerr going but its beyond me (just FYI, not looking for support), thats why I've gone this route.

The browser cache feature only works with a very few specific sites that are hardcoded into it. It's a nasty kludge solution that I really don't want to expand.

WebToEpub doesn't save story metadata (or even chapter text) in a consistent way from site to site. I did make some test code last year for reading WebToEpub output for ffnet, but it wouldn't work for other sites and I never came up with a way I liked to incorporate it into FFF's architecture.

For me, FlareSolverr 'just works' using the flaresolverr-v2.1.0-windows-x64.zip version and run from a command prompt. Until Cloudflare changes again, at least. That's always going to be a moving target.

[tricklem]

Originally Posted by tricklem

Just in case anyone else is or does have this issue.....I spoke too soon because I was still having this issue with FF.net, so I went into Preferences and under Adding Books unchecked Automatically Convert Books to the Preferred Output Format under adding Actions and under Automatic Adding. It seems to have fixed the issue as I just tried with updating a story from FF.net.

So, some additional info that I have learned about this to share with anyone also having this issue. I have also found that I have to delete the Original_EPUB file in the metadata before updating or I get the same issue. As soon as I delete that the update works like a charm. I am not sure how to stop Calibre from creating this. As soon as I delete it, then do my update (works no problem all chapters are there), then check the metadata again the file is back and I will have to delete it again before the next update. Totally weird.

I have attached a screen shot with the extra file circled.

EDIT: Just figured it out! So when I had to rebuild my library, there was an extra file in each file location labeled Original EPUB. I went in to the file location and deleted them all. Looks to have fixed the issue.

[JimmXinu]

New Test Version Posted

2021-12-21
- Refactoring of cli.py by hseg.
- Tweaks to base_xenforoforum tagsfromtitle default replace_metadata by hseg.