[GUI Plugin] FanFicFare

[JimmXinu]

Yeah, I'm calling that one a site bug.

We need to load the author page to get some of the metadata and without an author & author link, it's just not going to work.

[glitterbee]

Apologies if this has been answered and I just haven't found it, but: does FFF not read the kudos or hits counters on Ao3 anymore? I have both set up as integer columns which used to populate with kudos/hit count successfully, but at some point in the last few months (not sure when I noticed it, its been more a curiosity than anything) the Hits and Kudos counts were removed from the options in my custom columns menu. My personal.ini lists:

kudos=>#kudos
hits=>#hits

alongside other information. The text for "kudos=" and "hits=" appears in red.

I'm mostly confused since this configuration used to work. Am I doing something wrong, or did personal.ini formatting change with an update semi-recently?

[JimmXinu]

Originally Posted by glitterbee

I'm mostly confused since this configuration used to work. Am I doing something wrong, or did personal.ini formatting change with an update semi-recently?

kudos/hit count have never been available on the Custom Columns page drop downs--they are site specific. And the formatting of personal.ini hasn't changed.

Given you're seeing red high lights in personal.ini, I suspect you have a typo somewhere. Without seeing the whole thing, I can't really help.

If you want, there's a button for a 'Safe' version of personal.ini without user/password settings. You could post a copy here for us to look at.

[glitterbee]

Originally Posted by JimmXinu

If you want, there's a button for a 'Safe' version of personal.ini without user/password settings. You could post a copy here for us to look at.

personal.ini:

Spoiler:

[defaults]
is_adult:true

[epub]
include_images:true
jpg_quality: 100
keep_summary_html:true
make_firstimage_cover:true

[archiveofourown.org]
use_basic_cache:true
username:XXXXXXXX
password:XXXXXXXX
always_login:true
is_adult:true

extra_valid_entries:fandoms, freeformtags, freefromtags, ao3categories, comments, chapterslashtotal, chapterstotal, kudos, hits, bookmarks, collections, byline, bookmarked, bookmarktags, bookmarksummary, bookmarkprivate, bookmarkrec, restricted, series00, series00Url, series00HTML
fandoms_label:Fandoms
freeformtags_label:Freeform Tags
freefromtags_label:Freeform Tags
ao3categories_label:AO3 Categories
comments_label:Comments
chapterslashtotal_label:Chapters/Total Chapters
chapterstotal_label:Total Chapters
kudos_label:Kudos
hits_label:Hits
collections_label:Collections

bookmarks_label:Bookmarks
bookmarked_label:I Bookmarked Story
restricted_label:Restricted to Registered Users
series00HTML_label:Series
make_linkhtml_entries:series00,series01,series02,s eries03

replace_metadata:chapterstotal=>[0-9]*[^0-9]+[0-9]*=>

custom_columns_settings:
characters=>#characters
ships=>#relationships
rating=>#maturityrating
warnings=>#warnings
numChapters=>#currentchapters
numWords=>#wordcount
status=>#status
datePublished=>#dateposted
dateUpdated=>#dateupdated
fandoms=>#fandom
freeformtags=>#additionaltags
freefromtags=>#additionaltags
ao3categories=>#category
comments=>#comments
chapterslashtotal=>#chapterslashtotal
chapterstotal=>#totalchapters
kudos=>#kudos
hits=>#hits
bookmarks=>#bookmarks
collections=>#collection
restricted=>#restricted

bookmarked=>#bookmarked
bookmarktags=>#bookmarktags
bookmarksummary=>#bookmarksummary
bookmarkprivate=>#bookmarkprivate
bookmarkrec=>#bookmarkrec

series00=>#series1

[overrides]

Thank you for letting me know; I've removed the commented lines in the safe personal.ini for ease of navigation. I have kudos and hits in extra_valid_entries and have set up the appropriate custom columns (despite not being an option in the drop-down, I assumed I must have done that in the past, sorry about that) but they still will not populate anymore. Curiously, the custom_columns_settings all show up in red up to the "=", but the other columns don't have the same populating issue.

[JimmXinu]

The INI format continuation mechanism is to leave a space at the beginning of lines that should be included in the previous line--which includes *all* of the lines for custom_columns_settings.

Your custom_columns_settings, at least, need leading spaces on all the continuation lines. You didn't use [code] tag, so we can't see what leading white spaces you already have.

Like so:

Spoiler:

[defaults]
is_adult:true

[epub]
include_images:true
jpg_quality: 100
keep_summary_html:true
make_firstimage_cover:true

[archiveofourown.org]
use_basic_cache:true
username:XXXXXXXX
password:XXXXXXXX
always_login:true
is_adult:true

extra_valid_entries:fandoms, freeformtags, freefromtags, ao3categories, comments, chapterslashtotal, chapterstotal, kudos, hits, bookmarks, collections, byline, bookmarked, bookmarktags, bookmarksummary, bookmarkprivate, bookmarkrec, restricted, series00, series00Url, series00HTML
fandoms_label:Fandoms
freeformtags_label:Freeform Tags
freefromtags_label:Freeform Tags
ao3categories_label:AO3 Categories
comments_label:Comments
chapterslashtotal_label:Chapters/Total Chapters
chapterstotal_label:Total Chapters
kudos_label:Kudos
hits_label:Hits
collections_label:Collections

bookmarks_label:Bookmarks
bookmarked_label:I Bookmarked Story
restricted_label:Restricted to Registered Users
series00HTML_label:Series
make_linkhtml_entries:series00,series01,series02,s eries03

replace_metadata:
 chapterstotal=>[0-9]*[^0-9]+[0-9]*=>

custom_columns_settings:
 characters=>#characters
 ships=>#relationships
 rating=>#maturityrating
 warnings=>#warnings
 numChapters=>#currentchapters
 numWords=>#wordcount
 status=>#status
 datePublished=>#dateposted
 dateUpdated=>#dateupdated
 fandoms=>#fandom
 freeformtags=>#additionaltags
 freefromtags=>#additionaltags
 ao3categories=>#category
 comments=>#comments
 chapterslashtotal=>#chapterslashtotal
 chapterstotal=>#totalchapters
 kudos=>#kudos
 hits=>#hits
 bookmarks=>#bookmarks
 collections=>#collection
 restricted=>#restricted

 bookmarked=>#bookmarked
 bookmarktags=>#bookmarktags
 bookmarksummary=>#bookmarksummary
 bookmarkprivate=>#bookmarkprivate
 bookmarkrec=>#bookmarkrec

 series00=>#series1

[overrides]

[glitterbee]

Originally Posted by JimmXinu

The INI format continuation mechanism is to leave a space at the beginning of lines that should be included in the previous line--which includes *all* of the lines for custom_columns_settings.

Your custom_columns_settings, at least, need leading spaces on all the continuation lines. You didn't use [code] tag, so we can't see what leading white spaces you already have.

Like so:

Spoiler:

[defaults]
is_adult:true

[epub]
include_images:true
jpg_quality: 100
keep_summary_html:true
make_firstimage_cover:true

[archiveofourown.org]
use_basic_cache:true
username:XXXXXXXX
password:XXXXXXXX
always_login:true
is_adult:true

extra_valid_entries:fandoms, freeformtags, freefromtags, ao3categories, comments, chapterslashtotal, chapterstotal, kudos, hits, bookmarks, collections, byline, bookmarked, bookmarktags, bookmarksummary, bookmarkprivate, bookmarkrec, restricted, series00, series00Url, series00HTML
fandoms_label:Fandoms
freeformtags_label:Freeform Tags
freefromtags_label:Freeform Tags
ao3categories_label:AO3 Categories
comments_label:Comments
chapterslashtotal_label:Chapters/Total Chapters
chapterstotal_label:Total Chapters
kudos_label:Kudos
hits_label:Hits
collections_label:Collections

bookmarks_label:Bookmarks
bookmarked_label:I Bookmarked Story
restricted_label:Restricted to Registered Users
series00HTML_label:Series
make_linkhtml_entries:series00,series01,series02,s eries03

replace_metadata:
 chapterstotal=>[0-9]*[^0-9]+[0-9]*=>

custom_columns_settings:
 characters=>#characters
 ships=>#relationships
 rating=>#maturityrating
 warnings=>#warnings
 numChapters=>#currentchapters
 numWords=>#wordcount
 status=>#status
 datePublished=>#dateposted
 dateUpdated=>#dateupdated
 fandoms=>#fandom
 freeformtags=>#additionaltags
 freefromtags=>#additionaltags
 ao3categories=>#category
 comments=>#comments
 chapterslashtotal=>#chapterslashtotal
 chapterstotal=>#totalchapters
 kudos=>#kudos
 hits=>#hits
 bookmarks=>#bookmarks
 collections=>#collection
 restricted=>#restricted

 bookmarked=>#bookmarked
 bookmarktags=>#bookmarktags
 bookmarksummary=>#bookmarksummary
 bookmarkprivate=>#bookmarkprivate
 bookmarkrec=>#bookmarkrec

 series00=>#series1

[overrides]

Duly noted about the code tags. Thank you very much, this appears to have been the problem!

[FFlover99]

Hi, sorry if this has been asked before, but I couldn't find it and I've never used Calibre
I've decided to download my history from AO3, I spent 12h downloading it. After that, Calibre asked to update the library. When it was midway through, my computer cable got unplugged by accident.
I started downloading it again, 10 hours in the power goes out.
I tried finding the first donwloaded archives so I could just update the library with them , but I cannot find them for the life of me. Do they get deleted when the update is stopped? Thank you!

[JimmXinu]

First, I don't recommending doing large downloads like that. Both for the reasons you discovered, and the load it puts on the site's servers. It's been a while, but we've seen cases before of AO3 blocking user's computers for excessive traffic. Which is why I don't make any attempt to include a recovery feature.

But in the interests of using cycles already spent, I will tell you where to look.

FFF writes finished downloads to temp files in a temp directory inside the temp directory Calibre makes for each session. FFF removes its temp dir at the end of each download session and Calibre removes its temp dir when it quits.

The best way I know of to find out for sure where Calibre's temp dirs are on your computer is to run Calibre in Debug mode (under Preferences dropdown), do something that logs a temp file, like an FFF download or Convert a book to a different format, then look in the log.

For windows, each temp epub file typically looks something like this:

C:\Users\YOURUSER\AppData\Local\Temp\calibre_<randomletters>\fanficfare_<randomletters>\Story Title-Author-<randomletters>.epub

On my computer, nothing generally cleans up old calibre temp dirs if Calibre doesn't shutdown gracefully. So you may still find the downloads.

[FFlover99]

Originally Posted by JimmXinu

First, I don't recommending doing large downloads like that. Both for the reasons you discovered, and the load it puts on the site's servers. It's been a while, but we've seen cases before of AO3 blocking user's computers for excessive traffic. Which is why I don't make any attempt to include a recovery feature.

But in the interests of using cycles already spent, I will tell you where to look.

FFF writes finished downloads to temp files in a temp directory inside the temp directory Calibre makes for each session. FFF removes its temp dir at the end of each download session and Calibre removes its temp dir when it quits.

The best way I know of to find out for sure where Calibre's temp dirs are on your computer is to run Calibre in Debug mode (under Preferences dropdown), do something that logs a temp file, like an FFF download or Convert a book to a different format, then look in the log.

For windows, each temp epub file typically looks something like this:

C:\Users\YOURUSER\AppData\Local\Temp\calibre_<randomletters>\fanficfare_<randomletters>\Story Title-Author-<randomletters>.epub

On my computer, nothing generally cleans up old calibre temp dirs if Calibre doesn't shutdown gracefully. So you may still find the downloads.

Thank you so much! I found them all! I'm very grateful.
I didn't think of it like that, regarding the bulk downloading, but I'll certainly download small batches from now on. I'll be working on getting the metadata for the stories in the next week, so I'll do it low numbers at a time in intervals. Thank you for letting me know!
Have a great one

[bigwoof]

not sure what is going on with this particular book but page 20 of this fanfiction on fanfiction.net (using the browser cache feature on chrome) refuses to load.

FanFicFare just keeps re-opening the page over and over again in chrome and then just exits with a page load error (the epub produced shows an error on that page). it's the only story and page with this behaviour

I've deleted the browser cache and restarted the browser. I've deleted that page in the epub and updated. I've also asked FanFicFare to redownload the whole book. to no avail. everything else seems to work except this one page..

https://www.fanfiction.net/s/1429556...ightning-Thief

[JimmXinu]

I've heard of similar problems a couple times before, but we've never figured out exactly why it happens. AIR, as best we could determine, the chapter failing was marked non-cachable.

I was able to use browser cache successfully with this story on Firefox, but it looks like I do see the same problem with chapter 20 in Chrome.

I'll look into it, maybe I can figure something out this time.

[JimmXinu]

New Test Version Posted

2024-01-10
- Swap out SuperFastHash implementation #1026

Figured out the issue with the chapter and BrowserCache. I even opened an issue(#1026) to record it.

bigwoof, let me know if this fixes it for you.

[bigwoof]

Originally Posted by JimmXinu

New Test Version Posted

2024-01-10
- Swap out SuperFastHash implementation #1026

Figured out the issue with the chapter and BrowserCache. I even opened an issue(#1026) to record it.

bigwoof, let me know if this fixes it for you.

This has fixed the issue for me! Thanks! :-)

I also thought it was some caching flag in the header but could not find anything obvious.

A bug in the hashing function.. wow!! that's so rare...

I wonder if this will also fix the error I have with the browser cache on chrome where if the cache has been around for a while (say a few days), FanFicFare will just throw a "not in cache" error on new books even after you load them up in the cache (which is quite annoying after you spend the time to load up all the new chapters in the cache).

I work around this now by just clearing the cache before any big sync.. will monitor..

[JimmXinu]

Originally Posted by bigwoof

I wonder if this will also fix the error I have with the browser cache on chrome where if the cache has been around for a while (say a few days), FanFicFare will just throw a "not in cache" error on new books even after you load them up in the cache (which is quite annoying after you spend the time to load up all the new chapters in the cache).

Probably not?

Looking at the FFF debug output, Chrome doesn't update the 'creation' timestamp in the cache when clicking back and forth between chapters, on plain reload (ctrl-r), but it does on forced reload (ctrl-shft-r).

When clicking back and forth between chapters, Chrome's F12 networking debug shows a 200 OK (successful download) instead of a 304 Not Modified, which I see for plain reload (ctrl-r).

So I don't know what's going on there. Nor do I see any additional info in the cache I could use to better refine it.

What I can offer for that issue is: you can change FFF's browser_cache_age_limit setting. It's a float value of hours, -1 will use any found cache entry, regardless of age.

[bigwoof]

Originally Posted by JimmXinu

Looking at the FFF debug output, Chrome doesn't update the 'creation' timestamp in the cache when clicking back and forth between chapters, on plain reload (ctrl-r), but it does on forced reload (ctrl-shft-r).

When clicking back and forth between chapters, Chrome's F12 networking debug shows a 200 OK (successful download) instead of a 304 Not Modified, which I see for plain reload (ctrl-r).

Thanks for the explanation! :-)

Originally Posted by JimmXinu

What I can offer for that issue is: you can change FFF's browser_cache_age_limit setting. It's a float value of hours, -1 will use any found cache entry, regardless of age.

yeah, I'm already using that and it helps somewhat. some of the quirks I see are

1. the cache just stops working after a few days and FFF outputs a "file not in cache" error (forgot the actual error name) -- solution: clear the cache frequently before accessing the new pages you want FFF to download

2. some epubs (especially new single chapter epubs) *always* get reloaded by FFF even if you have a fresh cache and have access that epub and force-reload etc. -- solution: live with it. it just makes the whole download process a little longer and causes chrome windows to appear in the middle of your work :-)