[GUI Plugin] FanFicFare

[JimmXinu]

Originally Posted by hanjisung

Still having the issue of not being able to get URLs from PDFs, if anyone has advice on that front.

FFF doesn't know how to look inside PDFs. The "Search inside ebooks for Story URL?" option works on epub, html(technically zip) and txt.

Converting to one of those as culytera suggested may work. But it may not, depending on where there story URL is in the pdf. I'd try one or two before converting a ton.

[JimmXinu]

Originally Posted by vorm

Basically I created a plugin that intercepts the tab open command from Calibre and instead sends it to a Chrome extension that opens the page in the background and later closes the tab.

Well, that's a thing. I'm not going to reject this out of hand, but I'm also not going to immediately bless it.

Calibre Plugin

Monkey patching FFF's open_url() call from a different Calibre plugin seems a bit roundabout.

There's already a framework in FFF for calling a proxy. Combining fetcher_flaresolverr_proxy.py and cache_browser.py might have been easier.

Or if you've already gone to the effort of making a browser extension, why not make it a proxy that returns the requested page directly and skip using the cache?

Browser Extension

I'm honestly surprised a browser extension is allowed to open and listen to a port. Or is that only possible because you've put the browser in development mode?

BTW, if it doesn't already, this would be handy place to force uncached sites to save pages to cache. See https://github.com/JimmXinu/FanFicFare/issues/1312

Intentions

What is your(vorm) intention with this? Is this something you intend to support?

[vorm]

Originally Posted by JimmXinu

Well, that's a thing. I'm not going to reject this out of hand, but I'm also not going to immediately bless it.

Calibre Plugin

Monkey patching FFF's open_url() call from a different Calibre plugin seems a bit roundabout.

There's already a framework in FFF for calling a proxy. Combining fetcher_flaresolverr_proxy.py and cache_browser.py might have been easier.

Or if you've already gone to the effort of making a browser extension, why not make it a proxy that returns the requested page directly and skip using the cache?

Browser Extension

I'm honestly surprised a browser extension is allowed to open and listen to a port. Or is that only possible because you've put the browser in development mode?

BTW, if it doesn't already, this would be handy place to force uncached sites to save pages to cache. See https://github.com/JimmXinu/FanFicFare/issues/1312

Intentions

What is your(vorm) intention with this? Is this something you intend to support?

Fair enough on not blessing it. Quick answers to your points, plus why I built it this way.

Intention

I only archive every few months. Between checking for updates and new fics it can take a long time, and open_pages_in_browser keeps interrupting whatever else I’m doing, so I can’t leave it running in the background the way I used to. That was already annoying before, so I threw together a quick and dirty fix for myself.

It was meant to stay personal. I only shared it because I’d seen other people wanting something similar. I’m not planning ongoing support in any serious sense. I just posted a zip link; there’s no GitHub repo or anything. If people try it and it helps, great. If FFF changes and it breaks, that’s on me / them to notice.

Why monkey-patch from another plugin

I didn’t want to modify FanFicFare itself. Idea was: keep my hack working across FFF updates without re-applying patches every time. Hooking open_url from a separate Calibre plugin was the blunt way to do that.

I didn’t know about the existing proxy framework when I started, and it didn’t come to mind for the same reason: I was avoiding touching FFF. I’ll look at the Flaresolverr / proxy path you mentioned and see what I can do with that.

Why not “extension as proxy, skip cache”

Partly already there. The extension loads the page (fetch or one reused background tab), sends the HTML back to Calibre, and the plugin can feed that body to FFF when disk cache is missing/stale. Disk cache is still in play where it works; for no-cache / no-store type sites (same problem as #1312) the body handoff is what actually works. A cleaner proxy-style fetcher inside FFF would probably be better; this started from the browser-cache + open_url path because that’s what I was already using.

Extension listening on a port

It doesn’t. The Calibre plugin listens on localhost. The extension only polls / posts to that bridge. Developer mode is just so I can load the unpacked extension, not because of the networking.

#1312

Related: the extension rewrites cache headers so Chromium is more willing to store responses, and when that still isn’t enough it returns the page body directly so FFF isn’t stuck waiting on a disk-cache entry that never appears.

[culytera]

Question, wouldn't the default setting "use_view_full_work:true" be redundant on AO3 if you're logged in and you have "Show the whole work by default." enabled in preferences?

[vorm]

I looked at using the existing Flaresolverr proxy path for this. It works as a hook: point FlareSolverr_ProxyFetcher at a localhost helper, POST the URL, get HTML back from an extension. In that setup FFF does not call open_pages_in_browser, so silent loads work without monkey-patching open_url.

An MV3 extension cannot listen on a TCP port, so today something else must own localhost (my Calibre plugin, or a separate helper process) while the extension only polls and returns HTML.

If you want this to need only FFF + a browser extension, FFF would need to own that localhost side. For example: when a browser-helper mode is on, FFF (Calibre plugin and/or CLI) starts a small 127.0.0.1 server, accepts fetch jobs, and the extension long-polls for URLs and POSTs HTML back. FFF then uses that HTML like FlareSolverr_ProxyFetcher does. Protocol can be simple JSON or a thin Flaresolverr /v1 subset. FFF documents the contract; the extension stays out of tree.

Optional: when that mode is on, open_pages_in_browser uses the same path instead of safe_open_url.

That removes third-party Calibre plugins and standalone helpers. Stock Flaresolverr can stay as the headless option.

[hanjisung]

Originally Posted by JimmXinu

FFF doesn't know how to look inside PDFs. The "Search inside ebooks for Story URL?" option works on epub, html(technically zip) and txt.

Converting to one of those as culytera suggested may work. But it may not, depending on where there story URL is in the pdf. I'd try one or two before converting a ton.

Okay, thanks so much!

[JimmXinu]

Originally Posted by culytera

Question, wouldn't the default setting "use_view_full_work:true" be redundant on AO3 if you're logged in and you have "Show the whole work by default." enabled in preferences?

I did not know that was a thing. Yes, FFF would end up fetching the whole work twice.

I'll look into detecting when full work was already gotten.

[JimmXinu]

Originally Posted by vorm

It was meant to stay personal. I only shared it because I’d seen other people wanting something similar. I’m not planning ongoing support in any serious sense. I just posted a zip link; there’s no GitHub repo or anything. If people try it and it helps, great. If FFF changes and it breaks, that’s on me / them to notice.

Fair enough. I'm not sure I want to get into supporting a browser extension.

I also wonder whether the browser extension store gatekeepers would allow this.

Originally Posted by vorm

Extension listening on a port

It doesn’t. The Calibre plugin listens on localhost. The extension only polls / posts to that bridge. Developer mode is just so I can load the unpacked extension, not because of the networking.

That explains a lot. I was making faulty assumptions.

Originally Posted by vorm

#1312

Related: the extension rewrites cache headers so Chromium is more willing to store responses, and when that still isn’t enough it returns the page body directly so FFF isn’t stuck waiting on a disk-cache entry that never appears.

It looks like you're currently only patching open_url(), so I assume the 'returns page' part isn't actually used right now?

I'm not sure I see any cases where reading from the cache would be preferred if the page could be returned.

Flaresolverr has problems with returning images for reasons with how it's implemented. I would assume this does not?

Originally Posted by vorm

An MV3 extension cannot listen on a TCP port, so today something else must own localhost (my Calibre plugin, or a separate helper process) while the extension only polls and returns HTML.

If you want this to need only FFF + a browser extension, FFF would need to own that localhost side. For example: when a browser-helper mode is on, FFF (Calibre plugin and/or CLI) starts a small 127.0.0.1 server, accepts fetch jobs, and the extension long-polls for URLs and POSTs HTML back. FFF then uses that HTML like FlareSolverr_ProxyFetcher does. Protocol can be simple JSON or a thin Flaresolverr /v1 subset. FFF documents the contract; the extension stays out of tree.

That's what I thought. I considered this when we first started having problems with Cloudflare, but rejected it. I was still trying to maintain backwards compatibility to old python2 calibre versions at the time and I wasn't comfortable doing that threaded.

I will be thinking about this more.

[vorm]

Originally Posted by JimmXinu

Fair enough. I'm not sure I want to get into supporting a browser extension.

I also wonder whether the browser extension store gatekeepers would allow this.

Fair, unpacked/self-hosted is enough for a personal tool, I was not assuming Chrome Web Store distribution.

Originally Posted by JimmXinu

It looks like you're currently only patching open_url(), so I assume the 'returns page' part isn't actually used right now?

I'm not sure I see any cases where reading from the cache would be preferred if the page could be returned.

It is used. open_url is only the trigger on the browser-cache path. After the extension loads the URL it POSTs the body back. I also patch BaseBrowserCache.get_data so FFF reads that body first. Disk cache is fallback when there is no stored body. In practice the returned page is what makes no-store / stale cache cases work.

Originally Posted by JimmXinu

Flaresolverr has problems with returning images for reasons with how it's implemented. I would assume this does not?

From what I tested, images work no problem with this.

[JimmXinu]

New Test Version Posted

2026-07-24
- base_otw_adapter: Save and use if served full work initially
- base_otw_adapter: Save a network req if need to login.

If you check the 'Show the whole work by default.' setting in your AO3 account, you'll now save a network request any time you're downloading logged in. Thanks, culytera, for pointing that out.

Also saves loading that page until after login (if needed), saves reloading it after login.

[JimmXinu]

Originally Posted by vorm

Fair, unpacked/self-hosted is enough for a personal tool, I was not assuming Chrome Web Store distribution.

Whereas I don't want to explain installing it for naive users. And I want a firefox version.

Do we have any browser extension developers in the house? Even just to answer questions?

Originally Posted by vorm

It is used. open_url is only the trigger on the browser-cache path. After the extension loads the URL it POSTs the body back. I also patch BaseBrowserCache.get_data so FFF reads that body first. Disk cache is fallback when there is no stored body. In practice the returned page is what makes no-store / stale cache cases work.

Ah. I only looked for monkey patches in 'patcher.py'. Oh geez, there's what, 3?

That's part of the problem with monkey patching--it can happen anywhere. And the line between 'patch' and 'exploit' is entirely semantic.

I think if my motivation was not needing to update for new FFF versions, I would have faked being a flaresolverr server instead of all the monkey patching. But as you said, you came at it from browser cache.

IPC

How is the bridge server working with Calibre's BG job processes? Oh. It's using the bridge server both to talk to browser extension and all the FFF patches.

So there's nothing but convenience that says the bridge server has to live inside Calibre? Not that that isn't a good reason.

[vorm]

Originally Posted by JimmXinu

IPC

How is the bridge server working with Calibre's BG job processes? Oh. It's using the bridge server both to talk to browser extension and all the FFF patches.

So there's nothing but convenience that says the bridge server has to live inside Calibre? Not that that isn't a good reason.

Yeah, that is basically it. There is one localhost HTTP queue (the bridge). The extension polls that. Calibre BG download workers are a separate process, so they POST /open to the same GUI bridge and get the HTML back, they also need the open_url/page-store patches installed in-worker because the GUI patches are not visible there.

And yes nothing technical requires the bridge to live inside Calibre. That is mostly convenience (auto-start, shared settings, no extra process). A standalone helper with the same endpoints would work the same way, but i really didn't want to bother running calibre + server + extension to make this work, so adding to the plugin hack was the simplest path.

[culytera]

Originally Posted by JimmXinu

New Test Version Posted

2026-07-24
- base_otw_adapter: Save and use if served full work initially
- base_otw_adapter: Save a network req if need to login.

Also saves loading that page until after login (if needed), saves reloading it after login.

Thank you!

Another AO3-related question, when always_login isn't true, bookmarked and subscribed data aren't automatically set to False, they're simply missing. This isn't an issue for me personally since I have it set to true anyway.

However, when the cache settings are enabled and always_login isn't true, but the user's already logged in to their AO3 account on the browser, bookmarked and subscribed are still missing even when the work being downloaded has subscribed or bookmarked data. I know that always_login being false by default is to reduce network requests, but how is the behavior different once cache settings are enabled and a user's already logged in?

[JimmXinu]

Originally Posted by culytera

However, when the cache settings are enabled and always_login isn't true, but the user's already logged in to their AO3 account on the browser, bookmarked and subscribed are still missing even when the work being downloaded has subscribed or bookmarked data. I know that always_login being false by default is to reduce network requests, but how is the behavior different once cache settings are enabled and a user's already logged in?

Adapter deliberately only collects those if always_login is set.

 if self.getConfig("always_login") and LOGOUT_STR in data: # check actually is logged.
            # deliberately using always_login instead of checking for
            # actual login so we don't have a case where these show up
            # for a user only when they get user-restricted stories.

Browser cache feature did not exist when that was decided. AIR, collecting that data was an initial impetus for the always_login option.

Options include:

• Change to a new option (collect_my_stats?) that user has to set to collect that data.
  Requires existing users to add that setting(default on), or turn it off if they don't want those stats on restricted stories(default off).
  
• Remove always_login requirement.
  Collect those entries if logged in, don't if not. Without always_login, those entries only show up for user-restricted stories, other stories downloaded after a user-restricted story, or logged in Browser cache feature users. 'Random' inconsistent behavior is Bad.
  
• Leave as is -- Require always_login to collect that data.
  Browser cache suffers mildly for it (but only on get /series/?)
  Actually, if you assume BC and that you're already logged in browser, setting always_login:true and is_adult:false will save the extra req I think you're talking about.

So I think the answer is leave it as-is. Unless you think I've missed something?

[culytera]

Originally Posted by JimmXinu

• Remove always_login requirement.
  Collect those entries if logged in, don't if not. Without always_login, those entries only show up for user-restricted stories, other stories downloaded after a user-restricted story, or logged in Browser cache feature users. 'Random' inconsistent behavior is Bad.
  
• Leave as is -- Require always_login to collect that data.
  Browser cache suffers mildly for it (but only on get /series/?)
  Actually, if you assume BC and that you're already logged in browser, setting always_login:true and is_adult:false will save the extra req I think you're talking about.

So I think the answer is leave it as-is. Unless you think I've missed something?

To clarify what I mean by bookmarked and subscribed data, when always_login is true, I get either "True" or "False" for Bookmarked and Subscribed on all works. When always_login is false, bookmarked and subscribed values are simply missing even when trying to download works that should have either data. I understand this behavior without cache settings, but I thought this would be different once you're also using browser cache. I expected a work I'm subscribed to have the value subscribed:True since on the browser it clearly shows I'm subscribed to it, but because always_login is false, the value is just missing, not even a False even if incorrect.

It's not exactly a problem since I can just set always_login to true, but I was wondering why the behavior wasn't different if you're using browser cache. You did mention that this was before browser cache was a thing so that's likely why.

I don't have is_adult set to true since I'm always logged in and have the "Show me adult content without checking." in preferences checked so the behavior is as I expected.