[GUI Plugin] FanFicFare

[JimmXinu]

Originally Posted by hanjisung

Still having the issue of not being able to get URLs from PDFs, if anyone has advice on that front.

FFF doesn't know how to look inside PDFs. The "Search inside ebooks for Story URL?" option works on epub, html(technically zip) and txt.

Converting to one of those as culytera suggested may work. But it may not, depending on where there story URL is in the pdf. I'd try one or two before converting a ton.

[JimmXinu]

Originally Posted by vorm

Basically I created a plugin that intercepts the tab open command from Calibre and instead sends it to a Chrome extension that opens the page in the background and later closes the tab.

Well, that's a thing. I'm not going to reject this out of hand, but I'm also not going to immediately bless it.

Calibre Plugin

Monkey patching FFF's open_url() call from a different Calibre plugin seems a bit roundabout.

There's already a framework in FFF for calling a proxy. Combining fetcher_flaresolverr_proxy.py and cache_browser.py might have been easier.

Or if you've already gone to the effort of making a browser extension, why not make it a proxy that returns the requested page directly and skip using the cache?

Browser Extension

I'm honestly surprised a browser extension is allowed to open and listen to a port. Or is that only possible because you've put the browser in development mode?

BTW, if it doesn't already, this would be handy place to force uncached sites to save pages to cache. See https://github.com/JimmXinu/FanFicFare/issues/1312

Intentions

What is your(vorm) intention with this? Is this something you intend to support?

[vorm]

Originally Posted by JimmXinu

Well, that's a thing. I'm not going to reject this out of hand, but I'm also not going to immediately bless it.

Calibre Plugin

Monkey patching FFF's open_url() call from a different Calibre plugin seems a bit roundabout.

There's already a framework in FFF for calling a proxy. Combining fetcher_flaresolverr_proxy.py and cache_browser.py might have been easier.

Or if you've already gone to the effort of making a browser extension, why not make it a proxy that returns the requested page directly and skip using the cache?

Browser Extension

I'm honestly surprised a browser extension is allowed to open and listen to a port. Or is that only possible because you've put the browser in development mode?

BTW, if it doesn't already, this would be handy place to force uncached sites to save pages to cache. See https://github.com/JimmXinu/FanFicFare/issues/1312

Intentions

What is your(vorm) intention with this? Is this something you intend to support?

Fair enough on not blessing it. Quick answers to your points, plus why I built it this way.

Intention

I only archive every few months. Between checking for updates and new fics it can take a long time, and open_pages_in_browser keeps interrupting whatever else I’m doing, so I can’t leave it running in the background the way I used to. That was already annoying before, so I threw together a quick and dirty fix for myself.

It was meant to stay personal. I only shared it because I’d seen other people wanting something similar. I’m not planning ongoing support in any serious sense. I just posted a zip link; there’s no GitHub repo or anything. If people try it and it helps, great. If FFF changes and it breaks, that’s on me / them to notice.

Why monkey-patch from another plugin

I didn’t want to modify FanFicFare itself. Idea was: keep my hack working across FFF updates without re-applying patches every time. Hooking open_url from a separate Calibre plugin was the blunt way to do that.

I didn’t know about the existing proxy framework when I started, and it didn’t come to mind for the same reason: I was avoiding touching FFF. I’ll look at the Flaresolverr / proxy path you mentioned and see what I can do with that.

Why not “extension as proxy, skip cache”

Partly already there. The extension loads the page (fetch or one reused background tab), sends the HTML back to Calibre, and the plugin can feed that body to FFF when disk cache is missing/stale. Disk cache is still in play where it works; for no-cache / no-store type sites (same problem as #1312) the body handoff is what actually works. A cleaner proxy-style fetcher inside FFF would probably be better; this started from the browser-cache + open_url path because that’s what I was already using.

Extension listening on a port

It doesn’t. The Calibre plugin listens on localhost. The extension only polls / posts to that bridge. Developer mode is just so I can load the unpacked extension, not because of the networking.

#1312

Related: the extension rewrites cache headers so Chromium is more willing to store responses, and when that still isn’t enough it returns the page body directly so FFF isn’t stuck waiting on a disk-cache entry that never appears.

[culytera]

Question, wouldn't the default setting "use_view_full_work:true" be redundant on AO3 if you're logged in and you have "Show the whole work by default." enabled in preferences?

[vorm]

I looked at using the existing Flaresolverr proxy path for this. It works as a hook: point FlareSolverr_ProxyFetcher at a localhost helper, POST the URL, get HTML back from an extension. In that setup FFF does not call open_pages_in_browser, so silent loads work without monkey-patching open_url.

An MV3 extension cannot listen on a TCP port, so today something else must own localhost (my Calibre plugin, or a separate helper process) while the extension only polls and returns HTML.

If you want this to need only FFF + a browser extension, FFF would need to own that localhost side. For example: when a browser-helper mode is on, FFF (Calibre plugin and/or CLI) starts a small 127.0.0.1 server, accepts fetch jobs, and the extension long-polls for URLs and POSTs HTML back. FFF then uses that HTML like FlareSolverr_ProxyFetcher does. Protocol can be simple JSON or a thin Flaresolverr /v1 subset. FFF documents the contract; the extension stays out of tree.

Optional: when that mode is on, open_pages_in_browser uses the same path instead of safe_open_url.

That removes third-party Calibre plugins and standalone helpers. Stock Flaresolverr can stay as the headless option.