[GUI Plugin] FanFicFare

[ownedbycats]

I don't have a fanfic to test this with, but does FFF Calibre have any special handling for author names with ampersands?

[JimmXinu]

As I recall, there are some odd interactions with Calibre's code that automagically splits authors, but it depends at least partially on your Calibre tweaks settings.

I'll look into it more if you have a specific problem, but not just for speculation.

[ownedbycats]

Calibre splits via ampersand, yes. I was reading a fic (downloaded with another tool, and long-gone from the internet) where I had to change the author to replace ampersand with a plus sign, which made me wonder.

[Amalthia]

Originally Posted by JimmXinu

What about other stories that aren't restricted? Do they get genre while logged in?

Yes, the genre information is populated with non-restricted stories.

Originally Posted by JimmXinu

UPDATE: Another question: Do you see the Genre correctly in the title page inside the book? (Assuming you display it, it is by default.)

The genre information does not appear on the title page of the book with the rest of the metadata.

Amalthia

[Amalthia]

Originally Posted by JimmXinu

Amalthia, you might try running Calibre in Debug mode (under Preferences dropdown) while downloading and we'll see if there's any clues in the debug log.

I downloaded the story in debug mode and I've attached the log to this comment. I hope you the log makes more sense to you than it did to me.

Thanks again for your help on figuring this out.

Amalthia

[Amalthia]

Originally Posted by ownedbycats

another thought. from the default.ini:



do you get different results with 'a' rather than 'r'?

I tried "a" on your suggestion but it didn't make a difference.

[Amalthia]

Originally Posted by JimmXinu

What about other stories that aren't restricted? Do they get genre while logged in?

CORRECTION: It appears that the genre isn't populating for any new stories I'm downloading.

[Amalthia]

Originally Posted by JimmXinu

Amalthia, you might try running Calibre in Debug mode (under Preferences dropdown) while downloading and we'll see if there's any clues in the debug log.

The debug wasn't all that helpful, so I tested calibre portable and FFF with a limited amount of options in the personal.ini. The genre information loaded without any problems.

Something in my personal.ini is preventing the setting from loading. I figured I could add back a few settings at a time and see when the genre information fails to load. Or if you'd like I can share my "safe" settings with you and you may identify the problem faster than I could.

When/If I find the culprit I'll update my post with the solution.

[ownedbycats]

In your log, I noticed a problem with this replace_metadata:

FFF: INFO: 2023-01-05 20:10:08,787: story.py(813): Skipping replace_metadata line 'fandoms=>.*(陈情令 \|\ The Untamed|魔道祖师 - 墨香铜臭 \|\ Módào Zǔshī - Mòxiāng Tóngxiù).*=>\1\,陈情令 | The Untamed' on fandoms to prevent infinite recursion.

Try commenting it out (## at the beginning of the line) and see if it helps.

[JimmXinu]

Originally Posted by Amalthia

CORRECTION: It appears that the genre isn't populating for any new stories I'm downloading.

That makes a lot more sense than having something to do with AO3 restricted.

Originally Posted by Amalthia

Something in my personal.ini is preventing the setting from loading. I figured I could add back a few settings at a time and see when the genre information fails to load. Or if you'd like I can share my "safe" settings with you and you may identify the problem faster than I could.

I'll look at it if you'd like.

Originally Posted by ownedbycats

In your log, I noticed a problem with this replace_metadata:

FFF: INFO: 2023-01-05 20:10:08,787: story.py(813): Skipping replace_metadata line 'fandoms=>.*(陈情令 \|\ The Untamed|魔道祖师 - 墨香铜臭 \|\ Módào Zǔshī - Mòxiāng Tóngxiù).*=>\1\,陈情令 | The Untamed' on fandoms to prevent infinite recursion.

That's an issue, but I doubt it's this issue. However, I'm confused how this line is recursing. Most likely is if you have the same line twice?

You also have at least 5 broken plugins installed. But that shouldn't cause any problems other than a slight slow down on start up.

[Amalthia]

Hi,

I've tried everything and I'm at a loss. I'm attaching my safe personal.ini to this comment, plus a new debug version.

I've managed to fix it so that non-restricted stories are now downloading the "genre" information, however "update metadata" does not populate the column.

I'm using the "Custom Column" tab for "Genre" to rule out misspelling as an issue. In my bare bones basic portable version of FFF, the only information that is downloaded into "genre" custom column is the A03 category information like "M/F" nothing from additional tags.

I'm willing to setup a Zoom session to desktop share what my settings look like in case I'm missing something.

P.S. I've also noticed that genre information isn't downloading for Anthologies either...I'm not sure if the problem is related.

Again, thanks for any and all help in solving this mystery.
Amalthia

[JimmXinu]

New Experimental Version Posted (Here)

2023-01-06
- Refactor Fetcher & Browser Cache Code
- open_pages_in_browser Feature

This version contains a significant refactor of FFF's network fetching & caching layers plus a new browser cache feature.

1. In past, on FFF download start, the browser cache feature scanned all recent browser cache entries looking for ones that matched the few browser cache supported sites (fanfiction.net, fictionpress.com & ficbook.net).
   
2. This version instead looks for cache entries by hashed URL (like the browser itself does). This means that FFF will find updated/new cache entries instead of being limited to those found at start; but it also means the URLs have to match exactly.
   
3. This version changes the fanfiction.net adapter to include the story title in chapter URLs for that exact match. Epub update still seems fine.
   
4. It may be necessary (especially with ffnet) to reload the first viewed chapter page if CloudFlare intervened because it will have been cached with extra CF parameters.
   
5. I tried to keep support for reading the cache as filled by WebToEpub for ffnet working.
   
6. This version is not limited to a few hard-coded sites. However in my testing so far, many (most?) FFF supported sites explicitly serve story pages tagged "no-cache" and are not cached by the browser. Also, POST requests are not supported, so most (all?) logins are impossible.

This version introduces the open_pages_in_browser setting/feature.

If all of these settings are true:

browser_cache_path:...(your browser cache)...
use_browser_cache:true
use_browser_cache_only:true
open_pages_in_browser:true

...Then if FFF can't find a page in the browser cache, it will try to open it in your default browser as if you had clicked a link in a (non-browser) app, wait briefly, then look in the cache again. Sort of like a kludged proxy.

1. Obviously this will only work if you use the browser cache feature with your default browser.
   
2. This feature is very intrusive--in most OSes, opening a URL in browser like this will force your browser to the top and focus on it. You will almost certainly not be able to use this in the background while you do other things. The intended use cases are small numbers of updates, or as a last resort.
   
3. The open_pages_in_browser feature has retries, sleeps and 'quit trying after X fails' code that will probably need some tweaking. There's also currently a ton of debug output that will be reduced later.
   
4. Notably, the check_next_chapter:true ffnet adapter feature will often open pages to 'next' chapters that don't exist a couple times because the failure isn't cached. I'd recommend turning check_next_chapter off.

I've been using this refactored code since mid-December with few issues. Users who don't use browser cache shouldn't notice any difference.

Browser cache users who try this version without open_pages_in_browser may need to load/reload the page that CloudFlare intervened on more often. But hopefully that's it.

I will be interested to hear if users find open_pages_in_browser to be too intrusive or not. And if it works with other CF blocked sites.

New code branch: https://github.com/JimmXinu/FanFicFa...e/browsercache

[ownedbycats]

re: 1. Would it be difficult to have a browser_executable_path to provide the path to a non-default browser? If left empty, system default.

Since it's so obvious, I thought there's probably some reason it's not already in.

[JimmXinu]

It's not running a browser executable, it's calling the webbrowser package to pass a URL to the OS/window manager to be handled.

I haven't experimented with webbrowser.get('name'), no idea what--if anything--that will do.

[JimmXinu]

Originally Posted by Amalthia

I've managed to fix it so that non-restricted stories are now downloading the "genre" information, however "update metadata" does not populate the column.
...

I'm not seeing any issues with setting a tags-like column (from Custom columns setting tab--New Only unchecked) with Genre using your personal.ini and the story previously reported from a new download, an Update EPUB Always or from an Update from Saved Column.

If it appears on initial download, but doesn't update, I suspect the 'New Only' flag on the column, or possibly the 'Update Calibre Metadata' setting is unchecked (on each download/update dialog details, default on the General tab).

I'm not starting down the path of screen sharing to help individual users, sorry. This is a hobby, not a job.

Anthologies are a rather different beast and aren't as customizable as regular stories. I'd figure out the base case before worrying about anthologies. And accept that you won't have quite as much control.

There is an option to share all of your FFF options, not just the personal.ini, but it doesn't automatically protect your usernames and passwords. FFF Config -> Other(tab) -> View Library Preferences -> settings(left side list), then copy the text in the right side.