[GUI Plugin] FanFicFare

[JimmXinu]

Originally Posted by AbyssalAriel

I would have included an example of a URL having this problem (although really it's all of them from this site, both oneshots and series), but I believe I read somewhere that links to sites like literotica aren't allowed on the mobileread forum?

Correct, that is the exception to the general 'include story URLs' rule, thanks.

adapter_literotica doesn't get word count from the site. FFF will count words for you if you have turned on "Calculate Word Counts", then inject it into the metadata and re-write the output file.

FFF caches calculated metadata values for performance reasons. When numWords is set, it clears the cached value for numWords, but not for numWordsCopy. So that's why numWordsCopy isn't working for literotica.

I'm going to have to think about this a bit.

The 'easy' answer is to clear the cache when setting numWords, but that inflicts a sometimes sizeable performance penalty on all users.

Keeping track of all references in includes_in_ and conditional replace_metadata to clear cache for derived values would be a lot of new overhead code for an rather unusual corner case.

The only other idea that comes to mind is make the cache clearing user configurable for the few users who would want it, but that's kind of ugly.

[AbyssalAriel]

Originally Posted by JimmXinu

Correct, that is the exception to the general 'include story URLs' rule, thanks.

adapter_literotica doesn't get word count from the site. FFF will count words for you if you have turned on "Calculate Word Counts", then inject it into the metadata and re-write the output file.

FFF caches calculated metadata values for performance reasons. When numWords is set, it clears the cached value for numWords, but not for numWordsCopy. So that's why numWordsCopy isn't working for literotica.

I'm going to have to think about this a bit.

The 'easy' answer is to clear the cache when setting numWords, but that inflicts a sometimes sizeable performance penalty on all users.

Keeping track of all references in includes_in_ and conditional replace_metadata to clear cache for derived values would be a lot of new overhead code for an rather unusual corner case.

The only other idea that comes to mind is make the cache clearing user configurable for the few users who would want it, but that's kind of ugly.

Hm, that's a shame. Annoying, but I can live with it.

-----------

On another note, how do the cover generation options work, when used on existing books?

I'm finding it confusing because the "Update EPUB Cover?" option in the FFF updater doesn't appear to do anything. Depending on my settings in the "Calibre Cover" menu, the cover either gets updated even while the option is unticked, or it doesn't update whether or not I tick it.

And setting "Generate Calibre Cover: Yes, Always" will update and overwrite whatever cover was there before (which to be fair, makes sense), but FFF seems incapable of recovering the cover that "Yes, unless FanFicFare found a cover image" found even if I turn it back to "unless FFF found an image", unless I redownload the book (even though the same cover image is already inside).

[dailyalice]

Is it possible to just update the title page using Saved Metadata? I'm guessing not, after poking around. (Which is fine!)

In working on my guide, I was adding a section about how to customize the title page (by editing personal.ini) & thought I'd take a screenshot of what one looked like with default title_page settings. And then I realized it looked different than I would expect it to. And that was because I'd downloaded the work when I first started adjusting my library and working on the guide. So since then I've created all sorts of custom columns and moved things around. (Moving AO3 tags into their own column/field for instance, so those wouldn't show up in Genre as they did on the title page of the work I looked at.)

In the guide I keep telling people they can just update from Saved Metadata. And that works, of course, for metadata. I loved Saved Metadata!

And then I wondered if updating title pages from Saved Metadata was a thing. It doesn't appear to be a thing? And it's probably not necessary. But I thought I'd ask, in case I'm overlooking something. (Getting title pages to match up with how I'm organizing metadata in my library isn't all that important, certainly not worth redownloading works.)

A person could put some data like that on actual covers using generate cover, I guess too.

[ownedbycats]

Originally Posted by dailyalice

Is it possible to just update the title page using Saved Metadata? I'm guessing not, after poking around. (Which is fine!)

No. It's been requested in the past but Jim didn't think it fit. Personally, I just manually edit with the Calibre editor.

[JimmXinu]

Originally Posted by AbyssalAriel

On another note, how do the cover generation options work, when used on existing books?

AIR, FFF tends to keep the cover in the epub even if you have 'found an image' on because FFF assumes you put it there and want to keep it. Modify Epub plugin has an option to remove covers.

Cover generation is a bit of a mess due to the accretion of features over time. I don't use a fraction of it, nor do I keep all the details in my head.

Here's a link to the last time I can find it was discussed in detail.

If you have a specific case you can replicate it, please describe in detail the steps you take and the cover settings you are using, along with what you get and what you think you should be getting instead and I'll look at it.

Originally Posted by dailyalice

Is it possible to just update the title page using Saved Metadata? I'm guessing not, after poking around. (Which is fine!)

We were just discussing that, check the last few pages. Short answer is no.

I have come to believe the correct solution is:

Originally Posted by JimmXinu

If you are of a mind to write code, it might be worthwhile to consider either making a general purpose plugin for creating and injecting a metadata jacket (aka title page) from the existing Calibre data, or improving the existing metadata jacket feature in Calibre.

Then it could be used for all books, not just FFF output.

[dailyalice]

Originally Posted by JimmXinu

We were just discussing that, check the last few pages. Short answer is no.

Thank you. Also sorry about asking about something discussed so recently. I even recall skimming that! But at the time it wasn't relevant to what I was doing/looking for and apparently didn't register on some level. My brainpower is all being used on the guide project at the moment, I guess.

Yeah, a more general Calibre plug-in for adding title pages would be nifty.

Per other stuff being discussed, I've been using Modify ePub plug-in to replace covers for years and like it a lot for that purpose, when I've made new covers using Generate Cover or something and just want to stick a new cover into an existing ePub.

[AbyssalAriel]

Originally Posted by JimmXinu

If you have a specific case you can replicate it, please describe in detail the steps you take and the cover settings you are using, along with what you get and what you think you should be getting instead and I'll look at it.

1. Have a fanfiction that has no cover image on the site (or secondary images). Example: https://archiveofourown.org/works/37...pters/94371529

2. Generate Cover settings: one cover profile, for AO3, set in the menu (not in personal.ini), associated to archiveofourown.org

3. Settings in Calibre Cover tab:
Update EPUB cover from EPUB: Yes, if EPUB has a cover image
Generate Calibre Cover: Yes, unless FanficFare found a cover image
Generate covers only for new books: Disabled
Inject/update the cover inside EPUB: Enabled
Allow generate_cover_settings from personal.ini to override: True

4. Update the fic, with any setting. Update EPUB Always, or just updating from saved metadata.
Update Calibre Metadata: Enabled
Update EPUB Cover?: Disabled
Background Metadata: Disabled

Expected results: The cover stays the same, because "Update EPUB Cover?" is disabled.

Actual results: It generates a new cover with the plugin and injects that into the ePub.

-------------

1. A fanfiction with a cover image taken from the "first" image in the site. Example: https://archiveofourown.org/works/31387700

2-3: Same settings as above, except for:
Generate Calibre Cover: Yes, Always

4. Update the fic with any setting, and same options as above (including Update EPUB Cover?: Disabled)

Expected and actual results are the same as above. Cover updates even though it seems like it shouldn't.

5. Change back to "Generate Calibre Cover: Yes, unless FanficFare found a cover image"

6. Update with the same settings again, except "Update EPUB Cover?: Enabled"

Expected results: The cover changes back to the first image, since it's still there in the book.

Actual results: The cover image is stuck as it is. The only way to get the original image back is to use "Overwrite Always".

[AbyssalAriel]

Oh, did a bit more testing:

1-3: Same as the first situation.

4. Update the fic with "Update EPUB Always"

5. Change the plugin cover template for AO3.

6. Try step 4 again, but with "Update EPUB Cover?: Enabled"

Expected result: Updates the cover.

Actual result: Doesn't update the cover.

---

It seems like initially downloaded books that don't have a default or "first image" cover, don't have any "Cover_image" value shown in the saved metadata, and these ones can be changed perpetually even if I have "Update EPUB Cover?: Disabled" (which is undesirable). But they are set to "old" after running once with "Update EPUB Always", and end up stuck with that cover (even with "Update EPUB Cover?: Enabled") unless I use "Generate Calibre Cover: Yes, Always".

Sometimes, updating the books that already had a cover (from "first image") seems to change their "cover_image" value to "old" as well, for some reason (it looks like that happens if I use Update EPUB Always).

[AbyssalAriel]

So I was slightly wrong about the first situation; if using "Update EPUB Always", the cover doesn't change (regardless of the "Update EPUB Cover?" value), so you have to update from metadata to see it change even with "Update EPUB Cover?: Disabled".

I'd assume the same applies to "Update EPUB if New Chapters" in the case of there actually being new chapters, but I have to use "Always" because I can't really count on chapter updates while testing.

Using "Overwrite Always" resets everything back to the original state, with no "cover_image" metadata for the fics that don't have a cover or first image, or a value of "first" instead of "old".

[fdwojo]

Although my question is primarily for JimmXinu, I'll take an answer from anyone who can help:

I've liked downloading FF.net stories for the longest time, but with the crap going it, it's getting much harder. Calibre never did a great job for me, so I used to use a stand-alone program called FanFiction_Downloader, but even that is failing. So I'm now turning back to Calibre/FanFicFare.

Back on 8/13/2022, when you say you:

Originally Posted by JimmXinu

Honestly, I stopped using Flaresolverr and just use browser cache--along with basically not using ffnet for finding new stories anymore.

How do you make FanFicFare look in your browser cache? Because of my lack of using Calibre and FanFicFare that much, I'm kind of at a loss. Still, I don't want to have you waste a lot of time giving me an answer, so if you know a place I can look that describes a solution, that'd be good enough

Thanks anyone for any guidance. I feel like the person who never had to try this before, and I'm stumped, but I'd like to find a way to download my stories.

P.S. I've heard of FlareSolvErr, but aside from running it in the background, and not knowing if I have to give it any commands, it does nothing for me.

[dailyalice]

Looking for some clarification on backups. Hopefully this is an easy one to answer. Want to make sure i'm getting things right in that guide I'm working on.

The FanFicFare FAQ says this about backups:

How do I backup FanFicFare settings?
For CLI, back up your personal.ini file as you would any other file. You're already backing up your home directory and fanfiction library, right?

For Calibre plugin, FanFicFare keeps your settings in the Calibre library database--you backup your Calibre libraries, right?

Details of how to backup Calibre libraries are given in Calibre's User Guide - http://manual.calibre-ebook.com/faq....backup-calibre

And the Calibre FAQ which is linked to from there says:

The most important thing to backup is the calibre library folder, that contains all your books and metadata. This is the folder you chose for your calibre library when you ran calibre for the first time. You can get the path to the library folder by clicking the calibre icon on the main toolbar. You must backup this complete folder with all its files and sub-folders.

You can switch calibre to using a backed up library folder by simply clicking the calibre icon on the toolbar and choosing your backup library folder. A backed up library folder backs up your custom columns and saved searches as well as all your books and metadata.

If you want to backup the calibre configuration/plugins, you have to backup the config folder. You can find this config folder via Preferences → Miscellaneous. Note that restoring configuration folders is not officially supported, but should work in most cases. Just copy the contents of the backup folder into the current configuration folder to restore.

Are all of the FanFicFare settings in the actual Library folder? Or if you want to make sure you keep all your settings and tweaks and whatnot, do you need to also regularly copy the configuration folders? (It sounds like the latter in the Calibre FAQ, but the former in the FanFicFare FAQ.)

I do backup personal.ini textfiles regularly as a separate thing, especially before and after I make major changes to it.

I actually have been doing Calibre exports of my library thinking that is the best way to make sure I have a truly complete backup that covers everything, but I was wondering if that's overkill. Though it's easier than manually copying both library folders and config folders. (I've just been making exports into dated folders every so often.) (So currently my guide is telling people that exports are the best way to get a complete backup. Is that accurate? Am I misleading folks?)

Thanks! (Hopefully this hasn't been covered recently so I feel foolish again. I really do usually search all documentation and this forum before asking questions, but for this one I only got through documentation, not the forum as I'm getting burnt out from working on the Calibre/FanFicFare guide so much.)

[dailyalice]

Originally Posted by fdwojo

How do you make FanFicFare look in your browser cache? Because of my lack of using Calibre and FanFicFare that much, I'm kind of at a loss. Still, I don't want to have you waste a lot of time giving me an answer, so if you know a place I can look that describes a solution, that'd be good enough

Thanks anyone for any guidance. I feel like the person who never had to try this before, and I'm stumped, but I'd like to find a way to download my stories.

There's a page about Browser Cache on the FanFicFare Wiki:
https://github.com/JimmXinu/FanFicFa...erCacheFeature

And I know there's a lot about it in plugin-defaults.ini.

I haven't tried using it at all (so far) so have only just noticed these things in passing, can't advise on actual usage.

[fdwojo]

Well thanks for the link. I'll check it out, and maybe I'll get lucky.

I really need to be able to download the stories. I know I'm in a minority here, but I do NOT believe in reading stories I like online only. I always prefer to be able to download them, and unlike what Jinn said, I'm not willing to look elsewhere because the authors I like AREN'T elsewhere.

I'd hate to have to do it, but I might have to copy the stories, chapter by chapter and paste them into Sigil to make up the story EPUBs that I want. Time will tell.

[dailyalice]

I totally sympathize about frustration with FanFiction.net, when I was just getting back to using FanFicFare with Calibre, there were some stories I wanted that are only on FanFiction.net and I thought "well, I'll just go download those" only to realize it doesn't just work as it once did and will require a fair bit of messing around.

Probably a big part of why I just decided to focus on downloading from AO3 for now (and documenting how I do that). I threw up my hands at the FF.net mess, though I appreciate folks are working on it and appear to have come up with workarounds. (I just haven't delved into figuring out browser cache. Maybe it's easier than it looks to me at a glance.)

And yeah, I by far prefer downloading things to read them.

[JimmXinu]

Originally Posted by fdwojo

I really need to be able to download the stories.
...

We've all been dealing with this for close to 2 years now. If you're just now bumping into it, you should count yourself lucky.

The FFF FAQ on this issue points to some more resources.

If you don't already use Calibre, I honestly wouldn't recommend FFF with Calibre for ffnet anymore.

WebToEpub is a browser add-on, and there are several downloaders that use a built-in headless browser that work better now. BoFiLo and FanFictionDownloader are the ones I've heard mentioned lately, but I haven't used any of them recently.