[GUI Plugin] FanFicFare

[SilentShadow123]

Originally Posted by steltek

As an example for you, here is the setup of my personal.ini section for A03:

[www.archiveofourown.org]

 ##Anti-AI CSS setting follows:
use_workskin:true
 #slow_down_sleep_time:2
 #always_login:true
 #username:XXXXXX
 #password:XXXXXX
 #is_adult:true
use_browser_cache:true
 #browser_cache_age_limit:128.0
use_browser_cache_only:true
open_pages_in_browser:true

Any lines preceded by a # are remarked out and not used, but can be easily enabled if needed by editing the section to remove the # and saving the edited personal.ini . I've found I very rarely need the remarked options.

As I have it currently set up, FFF will only use the browser cache on the system default browser, and will open the pages in the system default web browser as needed. I do it this way as I've found this to be the most consistently stable way to do it over time.

You don't have to use the login/password options to log in to A03 for most downloads, but for problematic ones (i.e. padlocked stories) simply staying logged into your A03 account in a tab in your system default browser will almost always take care of any issues without having to mess with the login/password options.

In general, I recommend clearing the browser cache in your system default browser periodically (or when you experience problems). Further, if you do clear the cache, make sure to load a webpage in the default browser after clearing the cache (any random page will do) before attempting to use FFF. FFF will sometimes throw browser cache errors and not work when the browser cache is totally empty, and loading a page into the cache prevents this from happening.

The only other thing you need to do is make sure you have the correct browser cache path set. Windows and Mac tend to be fairly straightforward to locate based upon the FFF Wiki. Linux can be less so depending upon whether you are running your default browser from a browser actually installed directly in your Linux distro versus a browser running from a flatpack (i.e. the browser cache for flatpacked browsers is usually in a completely different location path than it is for installed browsers).

Thanks, I appreciate it. I've already managed to get the Browser Cache Method working for both Fanfiction.net and AO3 where I can see it actively collect the cache and open the tabs. It's made updating fanfics so much easier.

However, for AO3, I'm noticing a certain problem with the restricted works (blue lock) that I can't seem to figure out. I searched through, and I also don't see it addressed anywhere in past forum posts. Please feel free to redirect me to the solution if there is one.

Essentially, when I first open the fanfic url using FFF, it will navigate to the chapter index, and then to the chapter(s) I want downloaded. However, then FFF says that it can't be downloaded because it was "skipped" as, for example, it "already contains 1 chapters." I've also tried this with fanfics with multiple chapters, and the same thing happens where it says "already contains x chapters" where x is the total chapters for the fanfic. This is completely false, as I've failed to find the same fanfiction in my backups and calibre library. Do you happen to know why it's falsely saying this and skipping the download? Is there a bypass for this?

Edit: Using Overwrite Always fixes the issue, but if anyone can tell me, I'm still curious to know why this is happening when fetching cache for AO3 fanfics and if there's a fix. Thanks!

[steltek]

Originally Posted by SilentShadow123

Thanks, I appreciate it. I've already managed to get the Browser Cache Method working for both Fanfiction.net and AO3 where I can see it actively collect the cache and open the tabs. It's made updating fanfics so much easier.

However, for AO3, I'm noticing a certain problem with the restricted works (blue lock) that I can't seem to figure out. I searched through, and I also don't see it addressed anywhere in past forum posts. Please feel free to redirect me to the solution if there is one.

Essentially, when I first open the fanfic url using FFF, it will navigate to the chapter index, and then to the chapter(s) I want downloaded. However, then FFF says that it can't be downloaded because it was "skipped" as, for example, it "already contains 1 chapters." I've also tried this with fanfics with multiple chapters, and the same thing happens where it says "already contains x chapters" where x is the total chapters for the fanfic. This is completely false, as I've failed to find the same fanfiction in my backups and calibre library. Do you happen to know why it's falsely saying this and skipping the download? Is there a bypass for this?

Edit: Using Overwrite Always fixes the issue, but if anyone can tell me, I'm still curious to know why this is happening when fetching cache for AO3 fanfics and if there's a fix. Thanks!

In general, I keep FFF's output for the "Default If Story Already Exists" option set to "Update Epub Always". If I have a problem on AO3 that clearing the browser web cache and then closing/reopening the browser won't fix, I simply delete the fic and re-download it as it is usually far less effort to do so than trying to figure out what went wrong (especially when the problem that might have caused it is no longer happening on AO3).

And, as I previously stated, the simplest solution I've found for dealing with AO3's restricted stories is to ignore FFF's login options altogether and just keep a web browser tab logged into my AO3 account website at all times.

For this to work, though, you have to be logged in to your AO3 account in the default system browser being used to download into the browser cache. FFF should open what it needs for each fic separately in new tabs (worst case, generally 3 tabs at most per fic -- one for the fic navigation page, one for the first chapter of the fic, and a 3rd including the contents of the entire fic).

It is likely that missed chapters resulted from AO3 website issues -- while most of the big ones have been resolved for now, there are still random website glitches that occur - both Cloudflare related and not - all the time. FFF is usually very good about alerting you about fic downloads which have missing chapters before you update a fic.

And, finally, so far as I know FFF only updates the epub files it creates. If you have an epub in Calibre created or obtained from another source, FFF will not normally update it unless you do some sort of manual editing on it to make it compatible with FFF.

[DevilsReject97]

Originally Posted by JimmXinu

Maybe I will investigate adding an intermediate setting that keeps chapters that have succeeded, but still stops on first failure.

I wanted to see if you were still thinking about this? Or even an option to check box to keep them? Can't speak for others but I think it would be a huge help.

[JimmXinu]

Originally Posted by DevilsReject97

I wanted to see if you were still thinking about this? Or even an option to check box to keep them? Can't speak for others but I think it would be a huge help.

Already added to test version last month. (There hasn't been a new release since then.)

Originally Posted by JimmXinu

When continue_on_chapter_error was added, the use case was for old SB/SV story threads with manually crafted ToC posts that sometimes contained errors.

Today, we're seeing more and more sites that appear to start blocking after some unknown access limit is exceeded.

continue_on_chapter_error is helpful in those cases because you don't discard however many chapters did download before getting blocked. But I suspect that continuing to try each subsequent chapter after blocking starts will increase your blocked time--that's how I implemented it back in the day.

I propose to add a second setting: continue_on_chapter_error_try_limit.

FFF, (with continue_on_chapter_error:true), would record each chapter error and continue to the next chapter, but after continue_on_chapter_error_try_limit failures, would give up and mark all subsequent chapters 'ERROR' without trying to download them.

I'm thinking an appropriate default would be continue_on_chapter_error_try_limit:5

Some details:

• Why not just leave out the failed / not-tried chapters?
  Because users won't immediately notice missing chapters--I did exactly that while testing that very thing.
  
• What happens with updates?
  That's the real power of continue_on_chapter_error in this case; FFF will discard the ERROR message chapters and attempt to download them as if they were newly added. (Including be marked (new) if mark_new_chapters:true.)
  You could, for example, update a story with 1000 chapters once a day and if you get 100 or 200 chapters each time, have the whole thing in a week or so. Previously, we've instructed users to basically do that manually with chapter ranges.
  
• What about the original use case? What if there are actually that many bad URLs? How can you ever get the chapters after those?
  You'd need to increase continue_on_chapter_error_try_limit for that story, or use ignore_chapter_url_list.
  
• The ERROR chapter title would be the same, but ERROR chapter text for chapters after continue_on_chapter_error_try_limit would have different explanatory text.
  
• 'Skipping subsequent chapters without trying' does not apply to later chapters that were already in the epub. Those will still be used.

Opinions?

Originally Posted by JimmXinu

New Test Version Posted

2025-08-28
- Add continue_on_chapter_error_try_limit setting (default:5) See this post.

[MidnightRose]

I have no idea why, but this specific webnovel is producing a file that Calibre seems unable to parse.

https://www.royalroad.com/fiction/10...f-gold-digging

Trying to do any kind of conversion or using calibre's internal book editor gives this error:

calibre, version 8.11.0
ERROR: Failed to open book: Failed to open book, click "Show details" for more information.

Traceback (most recent call last):
  File "calibre\gui2\tweak_book\job.py", line 33, in run
  File "calibre\gui2\tweak_book\boss.py", line 101, in get_container
  File "calibre\ebooks\oeb\polish\container.py", line 1639, in get_container
  File "calibre\ebooks\oeb\polish\container.py", line 1232, in __init__
  File "calibre\ebooks\oeb\polish\container.py", line 279, in __init__
  File "calibre\ebooks\oeb\polish\container.py", line 282, in refresh_mime_map
  File "calibre\ebooks\oeb\polish\container.py", line 544, in opf_xpath
  File "calibre\ebooks\oeb\polish\container.py", line 649, in opf
  File "calibre\ebooks\oeb\polish\container.py", line 632, in parsed
  File "calibre\ebooks\oeb\polish\container.py", line 603, in parse
  File "calibre\ebooks\oeb\polish\container.py", line 179, in parse_xml
  File "calibre\utils\xml_parse.py", line 26, in safe_xml_fromstring
  File "src/lxml/etree.pyx", line 3428, in lxml.etree.fromstring
  File "src/lxml/parser.pxi", line 2059, in lxml.etree._parseMemoryDocument
  File "src/lxml/parser.pxi", line 1914, in lxml.etree._parseDoc
  File "src/lxml/parser.pxi", line 1930, in lxml.etree._parseDoc_unicode
  File "src/lxml/parser.pxi", line 1133, in lxml.etree._BaseParser._parseUnicodeDoc
  File "src/lxml/parser.pxi", line 647, in lxml.etree._ParserContext._handleParseResultDoc
  File "src/lxml/parser.pxi", line 765, in lxml.etree._handleParseResult
  File "src/lxml/parser.pxi", line 689, in lxml.etree._raiseParseError
  File "<string>", line 6
lxml.etree.XMLSyntaxError: internal error, line 6, column 3040

Meanwhile when I take the internal EPUB file and stick it into kindle previewer, it's able to open the file just fine.

Could it be the emoji in the description?

[MidnightRose]

After some experimentation, it is indeed the emoji in the description.
I tried downloading and importing the same book using WebToEpub, it imported fine, and could convert to other formats as well as edit using the epub editor.
Then if I update the metadata using FFF, it breaks again, but if I then remove just the emoji, it works fine again.
Perhaps an option to strip those out?

[kovidgoyal]

Just update calibre to 8.11.1

[JimmXinu]

Originally Posted by MidnightRose

After some experimentation, it is indeed the emoji in the description.
I tried downloading and importing the same book using WebToEpub, it imported fine, and could convert to other formats as well as edit using the epub editor.
Then if I update the metadata using FFF, it breaks again, but if I then remove just the emoji, it works fine again.
Perhaps an option to strip those out?

Originally Posted by kovidgoyal

Just update calibre to 8.11.1

Confirmed--Calibre 8.11.1 works and 8.11.0 doesn't.

[ownedbycats]

Jim: Anything in this change that might affect retrieval of collections?
https://archiveofourown.org/admin_posts/33031

[chrisric]

Is there a way to have FFF automatically use "Add New Book" for a novel that gets the
"Existing epub contains xxx chapters, web site only has yyy." message? Setting this at the
site level would be good. Per novel would be better but I think that would be a lot of work.

Why not just overwrite? Because the existing version may have chapters I haven't read yet.

Yes, I end up with multiple versions of the same novel.

Am I missing a better process for this kind of issue?

Thanks.

-Chris

[JimmXinu]

Originally Posted by ownedbycats

Jim: Anything in this change that might affect retrieval of collections?
https://archiveofourown.org/admin_posts/33031

Not that I'm aware of. But then I don't pay much attention to AO3 collections.

If you have stories that you know use collections perhaps you could try them?

Originally Posted by chrisric

Is there a way to have FFF automatically use "Add New Book" for a novel that gets the
"Existing epub contains xxx chapters, web site only has yyy." message? Setting this at the
site level would be good. Per novel would be better but I think that would be a lot of work.

Why not just overwrite? Because the existing version may have chapters I haven't read yet.

Yes, I end up with multiple versions of the same novel.

Am I missing a better process for this kind of issue?

When you ask for it be settable on per story basis, that sounds like some individual author is abusing chapters. I'm not a big fan of adding features to address individual authors.

Update mode is not settable by site or story. It's something the user can change on each download 'job', and can choose a default for.

Is this something other users would find valuable? I assume, from use pattern, that +99.9% you want the same setting. I'm honestly not sure how hard it would be to do.

As a work around:

1. Use a custom column to set FFF error to, say #fanficfare_error,
2. When you bump into one, search for #fanficfare_error:"Existing epub contains", make a saved search if you need it a lot,
3. FFF > Get story URLs from selected books (I have a hotkey for that),
4. The dialog has a copy to clipboard button, click it, ESC to close,
5. FFF > Actions by Update Mode > Download from URLs > Add New Book (set another hotkey to speed it up), hit Enter.

[LittleBit]

Originally Posted by JimmXinu

open_pages_in_browser_tries_limit is a recently added feature to keep from retrying forever. If you're using Chrome, you may need the latest test version--there was a change to caching.

Thanks! The test version did fix that problem.

So, I've been trying to simplify/automate a few things recently, but I'm stuck on this one. I have a Kindle that I'm using KOreader on and I have it set up to add to folders by fandom. Which is fine until I have crossovers and it generates a bunch of folders with like, 1 story in them. I want Calibre to have both fandoms listed but not use that column to send to KOreader. Is there a way to separate the fandoms into different columns, i.e 'MHA, Harry Potter' would into go into something like #fandom1 and #fandom2 columns, and I can just send #fandom1 to Koreader? Or have a second column that has all the single fandoms with crossovers converted to just a Crossover tag that would make a single folder in KOreader?

[JimmXinu]

Originally Posted by LittleBit

So, I've been trying to simplify/automate a few things recently, but I'm stuck on this one. I have a Kindle that I'm using KOreader on and I have it set up to add to folders by fandom. Which is fine until I have crossovers and it generates a bunch of folders with like, 1 story in them.

As a fellow KOReader user (on Kobo), I do it a different way: KOReader has a 'Calibre metadata search' feature you can use if you have a Calibre metadata.calibre file available. That works well for me because I put category(fandom) and genre in Calibre's Tags.

Whether you do or not have that available depends how you're getting your books onto the device.

(I just dump all my Calibre books(6700+) into one directory on the device. I used {id} to name them, but I'd use {uuid} if I did it again from scratch.)

Originally Posted by LittleBit

I want Calibre to have both fandoms listed but not use that column to send to KOreader. Is there a way to separate the fandoms into different columns, i.e 'MHA, Harry Potter' would into go into something like #fandom1 and #fandom2 columns, and I can just send #fandom1 to Koreader?

Yes, but it's complicated and depending on the site (coughAO3cough), one story can have hundreds of fandoms.

Originally Posted by LittleBit

Or have a second column that has all the single fandoms with crossovers converted to just a Crossover tag that would make a single folder in KOreader?

That would be easier. I'd suggest doing it as a Calibre Custom column instead of doing it in FFF. That way if you manually change the value in your multi-fandom column for a story, the single fandom column will update appropriately.