[GUI Plugin] FanFicFare

[watani]

So a problem I've encountered when getting Ao3 fics from the browser cache is that for very large fics where only a few chapters have been added since the last update, if I have 'open_pages_in_browser' active FFF does not give enough time for the entire work to be loaded in the browser before claiming that the update was bad. I instead have to restart the update and then it succeeds without issue because the full work page has now been successfully added to the browser cache.

To be more specific, the navigate and first chapter are opened in the browser before the "loading metadata" popup goes away, the job progress goes up to ~98%, the browser starts to open the full work but before it finishes loading FFF wraps up and says there was a bad update. The specific error shown is the following:

Error https://archiveofourown.org/works/XXXXXXXX Browser Cache Failed to Load with error 'unpack requires a buffer of 4 bytes'

Would it be possible to add a longer delay to give the browser time to load these larger fics instead of FFF failing immediately?

[JimmXinu]

New Test Version Posted

2025-02-24
- Fix for BrowserCache for images--cache partitioned by parent(story) page.

So, this is something people have reported before--images not loading from browser cache well. I didn't understand what was going wrong before because I wasn't seeing it / not looking hard enough.

Now that I'm having to use BrowserCache for AO3 too, I saw it and figured out the issue. I think.

FYI, it didn't show up for ffnet because ffnet doesn't allow images except their own.

[JimmXinu]

Originally Posted by watani

So a problem I've encountered when getting Ao3 fics from the browser cache is that for very large fics where only a few chapters have been added since the last update, ...

Story URL so I can see it too would be really nice...

Yes, I could make it longer. But after opening in browser, it already tries reading the cache 5 times with sleeps in between before repeating the whole cycle. I'd like to have something more concrete than 'longer'.

[JimmXinu]

New Test Version Posted

2025-02-24
- BrowserCache: Better handle cache file changing/failing while reading.
- Fix for BrowserCache for images--cache partitioned by parent(story) page.

Originally Posted by JimmXinu

Story URL so I can see it too would be really nice...

Yes, I could make it longer. But after opening in browser, it already tries reading the cache 5 times with sleeps in between before repeating the whole cycle. I'd like to have something more concrete than 'longer'.

Of course, immediately after this, I see the same problem when I go download my latest update. Just to make me feel like a jerk, I'm sure.

This is less about waiting and more about the file not finishing writing before we read it. I added better error handling and bumped up the wait times on the last 2 sleeps.

[RoundRobbinReadz]

Originally Posted by JimmXinu

I'm not sure what you mean, but I think you assume a greater degree of FFF-Calibre integration than exists.

I am new to using both Calibre and FFF, and I am learning! FFF is pretty darn awesome, so that could be lending itself to this impression. I am impressed by everything that it can do! Thanks for making it! (I hope this is an ok place to say thank you).

Perhaps, I am meddling with powers I can't possibly comprehend! Probably just a part of learning something new. But I may be in over my head.

On that note, thanks for the link to the wiki! So much info is contained there. It’s fabulous. I don’t want to bug you on here every time I have a question; Especially if there’s a place I can study to learn before asking a question here.

Originally Posted by JimmXinu

I refer to the data FFF collects and handles as "metadata entries", which are separate from Calibre data, AKA columns. Simplifying a bit, FFF metadata entries are divided into 'standard' entries collected for most/all sites and 'site specific' entries that are only used by that site. For example, every site has title and author; snapeflavours is potionsandsnitches.org only.

Ok, great. This is very helpful. I think I get this concept.

Originally Posted by JimmXinu

If you want FFF to populate a Calibre custom column on download/update, you can set it in the FFF Custom Columns config tab (for standard metadata entries) in personal.ini with custom_columns_settings.

Yep, this is the thing I am working on. Sorry for not being more specific with my first question. I included more specifics below.

Originally Posted by JimmXinu

If you want a Calibre custom column to appear in your ebooks' title page on update (it's assumed the Calibre book is created during initial download), there's a checkbox in the FFF personal.ini config tab 'Pass Calibre Columns into FanFicFare on Update/Overwrite' and a button to show you want what FFF metadata entry names each column will be placed in. They will automatically be labeled the same as the Calibre Columns.

Yes! I love this feature! I found it 2 weeks ago and it made everything better. My columns are beautifully populated. Before, I was just trying to have a custom column show Ships. Felt like a manageable goal at the time, but I wasn’t able to make it happen. Selecting this checkbox in the FFF personal.ini config tab made it happen. Clicking the box for 'Pass Calibre Columns into FanFicFare on Update/Overwrite', changed everything! Gave me hope to try more things with FFF. Prior to this, I was trying to do things with metadata entries and basically seeing no results.

I want to figure out how to populate a custom column with either type of metadata entries, standard & site specific entries. I can probably figure this out from the wiki, now that I know where to look.

But I think part of where my problem is, and/or is indicated here:
Under the FFF config tab for Custom Columns, I see the name of the column I made on the left, but don't know what metadata value type to fill it with. The name of the column isn’t listed as an option in the menu. I presume this is related to it not being standard data entries, or if site specific, the personal.INI is incorrect.

However, it’s listed in the drop down menu under Special Column, Update/Override Error Column has several of my column names listed, even ones that seem to be populating?

Example: I have a column for Fandom, set to seek #Fandom, but the metadata value type I chose is Category. It seems to be working.

Originally Posted by JimmXinu

You can then used those metadata entry names in titlepage_entries or logpage_entries.

OK. I haven’t changed anything under the section for Titlepage_entries. Seemed like updating the Title Page was a good future goal.

Originally Posted by JimmXinu

The FFF github project has a wiki which also has an FAQ.

A lot of INI settings are documented primarily in defaults.ini. You can see the defaults.ini for FFF Calibre plugin from the button on the FFF personal.ini config tab , or on github.

Thank you for all of these! I am glad to discover the wiki.


I was trying the idea discussed in this thread. (Post numbers #8111, #10105 and #10106.)

https://www.mobileread.com/forums/sh...postcount=8111
https://www.mobileread.com/forums/sh...ostcount=10105

https://www.mobileread.com/forums/sh...ostcount=10106

This idea seemed smart. Have a column with all the Tags, and a column for Tags I edited (cleaned up) before I start learning anything Replace metadata. But, this may not be necessary at this time. I figured it was a good thing to explore next - cleaning up tags.
I tried working with this code, from #10106 under the archiveofourown section of personal INI. So far, I’m stuck. But now, I have the wiki to explore. It may hold the info I need to GROK!

[ChaoticAdventure]

Originally Posted by fishing_forever

I've been trying to use the workaround and set the browser_cache_path in defaults. It opens the fic link in my default browser (Firefox), but then I get this error:

HTTP Error in FFF 'Page not found or expired in Browser Cache (see FFF setting browser_cache_age_limit)'(428)

I tried setting the browser_cache_age_limit to -1 but that didn't do anything.

When I try to update the same fic again with FFF, it doesn't open it in my browser a second time.

I am also having this issue with FireFox. I don't suppose you found a solution for it?

[Amalthia]

Originally Posted by ChaoticAdventure

I am also having this issue with FireFox. I don't suppose you found a solution for it?

I'm having similar problems when I try and update series. I don't know why but the caching from Ao3 doesn't seem to work on series.

I'm going to try again in a few days to update a few of the series I'm following.

I'm also not updating series every time there is an update. A few writers seem to have an update daily...which I guess I shouldn't complain about too much because at least they are updating their story.

[fishing_forever]

Originally Posted by ChaoticAdventure

I am also having this issue with FireFox. I don't suppose you found a solution for it?

Unfortunately not, but I found that if a fic is already in your Calibre library and there aren't additional chapters, you are able to update it. It will say "Already contains [number] chapters" if you select Update EPUB if New Chapters or "Update epub completed, added 0 chapters for [number] total" if you say Update EPUB Always.

However, the problem comes up if you try to overwrite an existing fic or if you are trying to download a new fic/new chapters of a fic.

[JimmXinu]

Originally Posted by RoundRobbinReadz

But I think part of where my problem is, and/or is indicated here:
Under the FFF config tab for Custom Columns, I see the name of the column I made on the left, but don't know what metadata value type to fill it with. The name of the column isn’t listed as an option in the menu. I presume this is related to it not being standard data entries, or if site specific, the personal.INI is incorrect.

Only the standard metadata entries are listed in the Custom Column dropdowns, and only for column types that make sense.

For example, a numeric column will only offer you Words and Chapters because those are the only ones guaranteed to be numbers.

Site specific metadata entries have to be set using custom_columns_settings.

Re: AO3 Issues (unrelated to RoundRobbinReadz' post)

I'm doing what I can, when I can, to address AO3's on-going issues.

Basically, their servers are having problems. Part of addressing that is more aggressive (cloudflare, etc) blocking of automated access. See https://archiveofourown.org/admin_posts/31204

It doesn't help that Mac (at least) has started blocking open_pages_in_browser for at least some users in a recent update. I don't have a fix for that as yet, and I don't know if there will be one.

Unfortunately, I don't have the bandwidth right now to help each individual user sort out their exact configuration issues with BrowserCache. Hopefully people can help each other.

[ownedbycats]

Is it strange that I haven't had any of the reported issues with AO3? Maybe just lucky.

[kolbinsky]

Hi am just wondering is there a way to bypass the amount of chapters found when updating a fanfic from fanfiction.net since i am trying to update a fanfic and it says Existing epub contains 887 chapters, web site only has 857. Use Overwrite or force_update_epub_always to force update.
I want it to add the new chapter.
I dont want to overwrite because i have to click every chatper in the story again and i tried Update Epub Always but it still shows that chapters are missing when i update the story. This is what it says
Status: Bad
Title: Rock n' Not Loud Adventures
Author: heavy5commando
Comment: Existing epub contains 887 chapters, web site only has 857. Use Overwrite or force_update_epub_always to force update.
URL: https://www.fanfiction.net/s/1413969...oud-Adventures

[JimmXinu]

Basically, the site is broken for that story.

When I look , I see chapter 1 lists 857 chapters, chapter 857 lists 866. And so on.

You can try using the setting check_next_chapter:true, which--if you see the same thing I do--in this case will look at chapter 858 and happens to list 888 chapters.

So you may get lucky. But broken site is broken.

[Amalthia]

What I find interesting with the Ao3 stuff is I am still having better luck downloading when I don't use the cache. Sometimes just coming back a few hours later works.

[ownedbycats]

Originally Posted by JimmXinu

Basically, the site is broken for that story.

When I look , I see chapter 1 lists 857 chapters, chapter 857 lists 866. And so on.

You can try using the setting check_next_chapter:true, which--if you see the same thing I do--in this case will look at chapter 858 and happens to list 888 chapters.

So you may get lucky. But broken site is broken.

I've noticed FanFiction.net being very unreliable with newer updates. Like disappearing then reappearing later, or 'stoey does not have any chapters" even though it does. Others report it too: [1] [2] [3]

[JimmXinu]

New Test Version Posted

2025-02-28
- base_xenforoforum: Add details_spoiler option for #1165
- Add [base_otw] with use_basic_cache:true to defaults.ini