[GUI Plugin] FanFicFare

[ratspootin]

I'm trying to bulk download AZW3 from AO3 but the only option I'm seeing with FFF is MOBI, EPUB, HTML and TXT, presumably because AO3 is one of the few (if only) sites that have AZW3 as a format.

Is there an option already built in to FFF to enable AZW3 that I'm missing? And if not, I'm assuming the solution would be to configure my personal ini file to force AZW3 format when downloading from archiveofourown.org but I wouldn't even begin to know what that code is.

My goal is to get around Send to Kindle stripping the TOC out of AO3's EPUB files, and converting AO3's AZW3 to EPUB works beautifully (unfortunately MOBI to EPUB makes an even bigger mess, while EPUB to EPUB doesn't change anything). Naturally, the one format I don't have saved locally is the one I need.

[JimmXinu]

I believe you are misunderstanding how FanFicFare(FFF) works and what it's for.

FFF doesn't download any of AO3's ebook formats, FFF downloads HTML and generates its own ebook format files.

The recommended output format for FFF is EPUB because that is the one format that FFF supports updating. The other output options are honestly only there because they already were in the code base when I came to project.

The primary use case for FFF (IMHO, at least) is to download updates to stories you are following with the first secondary use case being downloading new stories.

I read on Kindle myself and use Calibre's Convert feature to make AZW3s from the EPUBs and side load them using USB.

Although you apparently want to convert AZW3 to to EPUB anyway? I don't know what all that is about.

You mentioned 'bulk download'--I generally recommend against doing large numbers of downloads at once from any given site because many of them have mechanisms that will detect that and throttle or block your access to the site.

[ratspootin]

Originally Posted by JimmXinu

FFF doesn't download any of AO3's ebook formats, FFF downloads HTML and generates its own ebook format files.

Thanks Jimm! FFF converting from HTML actually clears things up. I've only use FFF to get updated metadata for fics I downloaded directly from AO3, so my understanding around the output format functionality was very much off.

The issue is with Amazon dropping support for MOBI. Send to Kindle does not like how AO3 formats their EPUBs so when the file arrives on your Kindle, the table of contents are missing. For whatever reason, if I download an AZW3 from AO3's website, and convert it to an EPUB with Calibre, this problem doesn't happen.

For now I'm just continuing to download MOBI from AO3, but eventually Send to Kindle is going to drop support for MOBI entirely and I'm trying to figure out a solution that doesn't require a USB cable before that happens.

[ownedbycats]

Unless you particularly prefer how AO3 formats the eBooks, maybe see whether email to Kindle likes the FFF-downloaded epubs better.

[JDesat]

I'm not sure if this is a new issue or if it's only happening to me (I scrolled through the last few pages of this thread and didn't see anything about it), but in the last few days, maybe a week, I've been having new issues with downloading stories from FFN.

If it's a new story I'm downloading I tend to use WebToEpub to cache the pages, though I've noticed this error happening even when I load them manually (first and last chapter when it's just an update).

When using WebToEpub any chapter where the plugin has halted and I've had to manually jump to that page to get it going again doesn't get downloaded (even though, as far as I know, it should be doubly cached, once from my manual loading of the page and once through WebToEpub). I've tried throttling the speed at which WebToEpub runs, but even when those errors/pauses don't pop up I find I'm still missing at least one chapter. Further, if I try WtoE once and it halts at times, if I throttle it, run it again, and then try and download with FFF again the same chapters are still missing.

Sadly, I didn't even think to check what chapter(s) are missing when I have had the issue on stories that are just updates (though I assume it has to be the new chapter since that's the only one that should be downloaded). In either case, and there might be a workaround for this that I don't know, I've been deleting the story from Calibre, and generally running WebToEpub again later and then FFF to download it as a fresh story. This has normally worked - though there is one story that I've been unable to download in its entirety for the last few days and it's been updating daily: https://www.fanfiction.net/s/1411117...-of-the-Desert).

I'm sure this is a rambling mess, but if there's either a way to avoid this entirely, or a way to just have FFF download and add the missing chapters, I'd love to hear it. I'm wondering if it's an issue with how the pages are getting cached on my end, but I haven't changed anything. I think I'm going to try waiting for the 4 hours (I think?) the pages are cached to go by, then run WebToEpub throttled so it doesn't pause at all, then try FFF again. I haven't tried that yet. But even if that works, if there's a setting I can use to download/fix missing chapters I'd like to know.

Thanks!

[ownedbycats]

On Vivaldi (based on Chromium), I occasionally need to deliberately empty out the cache before re-loading the pages in or FFF has trouble. Is this something you tried already?

[JimmXinu]

Originally Posted by JDesat

...
When using WebToEpub any chapter where the plugin has halted and I've had to manually jump to that page to get it going again doesn't get downloaded (even though, as far as I know, it should be doubly cached, once from my manual loading of the page and once through WebToEpub).
...

If you are using the released version of FFF, I don't know.

If you are using the test version, this could be a consequence of the browser cache feature changes.

If you see the cloudflare screen before a chapter, it is probable that chapter was fetched with a POST or with additional GET parameters, which means it won't be cached where FFF looks for it.

Load the page by clicking in the URL box and hitting Enter (instead of just hitting reload), or click the story 'Prev' button and then click the story 'Prev' button (not the browser back/next buttons) to load it explicitly.

And you might need to force reload, not just reload.

Most browsers allow you to 'force reload' a page by holding the shift or control key while clicking or pressing reload(aka refresh). Which exactly varies by browser--Google will be able to tell you. This will update the browser cache for that page.

[JDesat]

I'm using the test version. I'll try the options you've given as well.

@JimmXinu If it isn't happening to anyone else, don't worry about it. It's mainly the one story (everything else I've been able to get clean, full copies of even if it took a few attempts), and it's not even a story I've started reading yet. I made the post to let you know it was happening to me and see if it was happening to anyone else. If it's just me, don't waste your time looking for a fix! You do enough work on this amazing Calibre plugin already, I don't want to pile more on your plate!

Also, I might just be an idiot... For the longest time when I've had a pause/halt in WebToEpub I've jumped to that chapter of the fic and it always seemed to speed things up so the plugin cached pages again more quickly. Just now I let it run (3 second throttle) and it DID pause, but rather than do anything myself I just waited and eventually it resumed... then FFF was able to download the story fine. I'll try some patience from now on and see if that works too.

[JimmXinu]

New Test Version Posted

2023-01-23
- adapter_deviantartcom date changes #910, thanks bugmaschine
- Revamp retries for browser cache with open_pages_in_browser

Now shares code with royalroad.com and novelall.com adapters for relative dates. Eg, '2 days ago'.

[ownedbycats]

I downloaded this ffnet story with browser cache. I opened the cover image to load it into cache, but got a generated cover anyways.

User error?

[JimmXinu]

Originally Posted by ownedbycats

I downloaded this ffnet story with browser cache. I opened the cover image to load it into cache, but got a generated cover anyways.

User error?

Probably? I got it using open_pages_in_browser:true.

[ownedbycats]

Question: How do conditionals handle replaced values?

Example:
category=>foo=>bar

Would I use &&category==foo or &&category==bar?

[chrnno]

Originally Posted by ownedbycats

Question: How do conditionals handle replaced values?

Example:
category=>foo=>bar

Would I use &&category==foo or &&category==bar?

Replace metadata does lines one-by-one in order and on conditionals simply checks what the value is at that point. Does not keep track if value has been changed previously, nor skip ahead to check if it will be changed.

[JimmXinu]

Originally Posted by ownedbycats

Question: How do conditionals handle replaced values?

Example:
category=>foo=>bar

Would I use &&category==foo or &&category==bar?

Originally Posted by chrnno

Replace metadata does lines one-by-one in order and on conditionals simply checks what the value is at that point. Does not keep track if value has been changed previously, nor skip ahead to check if it will be changed.

Close, but not quite. Yes, it does line-by-line, but it does it on the value currently requested. What happens with a conditional is that the same look code is recursively called on the conditional entry. (Presumably 'bar' in this case.)

In this case, category when it's found in &&category==bar and the final value of category after all it's transformations is what's used.

But to prevent infinite recursion, nothing that changes category can use category for a conditional. It's easy do that accidentally and not realize it.

[ownedbycats]

That works. In this case it was a replacement on a character.

Also, I just realize the character name is formatted this way only on FanFiction.net ("Firstname L." rather than "Firstname Lastname")), so I'm putting it in that section. Do I safely assume that add_to_replace_metadata always runs after replace_metadata?