[GUI Plugin] FanFicFare

[JimmXinu]

New Version Released

Version 4.43.0 - 2025-03-01
- base_xenforoforum: Add details_spoiler option for [#1165](https://github.com/JimmXinu/FanFicFare/issues/1165)
- Add [base_otw] with use_basic_cache:true to defaults.ini
- BrowserCache: Better handle cache file changing/failing while reading.
- Fix for BrowserCache for images--cache partitioned by parent(story) page.
- base_otw_adapter: Detect & report 'This site is in beta' page
- AO3: Double default slow_down_sleep_time
- For adapter_mcstoriescom: Remove /../ from Get Story URLs from web page
- adapter_mcstoriescom: Suppress site URLs that look like stories but aren't. #1160
- adapter_fanfictionnet: Attempt chapter from m. (vs www) when chapter not found
- adapter_literotica: Get more story urls. #1159 Thanks, dbhmw
- Add include_tocpage:always option.
- adapter_fimfictionnet: Further cover fix
- adapter_fimfictionnet: Fix cover images and use data-source attr for img src.
- adapter_storiesonlinenet: Remove some code that broke parsing when 'author' was in the title.
- adapter_literotica: http->https
- adapter_literotica: Site changes for non-www domains.
- Make plugin use own copy of six only--including in Smarten Punc
- adapter_fictionlive: fix bounds check in vote accumulaton. resolves JimmXinu#1154, Thanks HazelSh
- Make plugin use own copy of six only.

[vikarti]

Strange issue.
DeviantArt sometimes doesn't like downloading too much too fast, this results in 403 errors.
slow_down_sleep_time fixes this if I just use regular download but it does NOT fix this if I use 'Get Story URLs'(only one story gets downloaded, others get 403s). It looks like slow_down_sleep_time is not being applied when 'Get Story URLs' used.

[JimmXinu]

Originally Posted by vikarti

Strange issue.
DeviantArt sometimes doesn't like downloading too much too fast, this results in 403 errors.
slow_down_sleep_time fixes this if I just use regular download but it does NOT fix this if I use 'Get Story URLs'(only one story gets downloaded, others get 403s). It looks like slow_down_sleep_time is not being applied when 'Get Story URLs' used.

Can you share a URL you see this happening with? Use PM if NSFW, please.

[ownedbycats]

ffnet question How do I get a new chapter when ffnet is fucking up and it doesn't appear in the TOC for chapter 1? I tried using chapter page directly but it didn't work and says only 74 chaps. Is there a way to override where it gets the links?

https://www.fanfiction.net/s/1288375...-Transcendence





EDIT: And Chapter 75 is gone again. What is wrong with this site.

[Clem2605]

Hi!

I've begun tu use the browser cache/open_in_browser features with AO3 at least temporarily,because I was having way too many 525 errors without it, and I noticed something.

I use the option to download from Entire Works, and when opening the work in my browser, FFF opens both the navigation page of the work, and the work itself.

I'm assuming that FFF opens the navigation page to grab the links to all the chapters, but when downloading from entire works, this probably becomes redundant since FFF doesn't open those pages individually anymore.

Could it be possible to stop opening this page when the option to download from entire works is enabled?
This probably does more than I suspect, or maybe this isn't possible, but it would reduce by half the traffic to AO3 for each query if doable, so I thought it would be worth it to bring it up.

[JimmXinu]

Multi-reply

Originally Posted by ownedbycats

ffnet question How do I get a new chapter when ffnet is fucking up and it doesn't appear in the TOC for chapter 1? I tried using chapter page directly but it didn't work and says only 74 chaps. Is there a way to override where it gets the links?

You can try check_next_chapter and meta_from_last_chapter.

## ffnet used to have a tendency to send out update notices in email
## before all their servers were showing the update on the first
## chapter.  It generates another server request and doesn't seem to
## be needed lately, so now default it to off.
check_next_chapter:false

## ffnet sometimes doesn't update the metadata on the first chapter
## immediately.  meta_from_last_chapter:true will instruct the adapter
## to take the metadata from the last chapter instead of chapter 1.
## It can generate another server request and usually isn't necessary,
## so it defaults to off.
meta_from_last_chapter:false

Beyond that, broken site is broken.

Originally Posted by Clem2605

I'm assuming that FFF opens the navigation page to grab the links to all the chapters, but when downloading from entire works, this probably becomes redundant since FFF doesn't open those pages individually anymore.

Could it be possible to stop opening this page when the option to download from entire works is enabled?

Short answer is no. I'm not interested in changing FFF's paradigm to save one req on one type of site. Plus there's info on the nav page not available elsewhere.

[ownedbycats]

Another ffnet question: outdated metadata because of brokenness. Is there a way to use count pages for wordcount on that site, but listed wordcount for other sites? It's not a big deal.

EDIT: Actually I compared the chapter notification emails and luckily the wordcounts are updating. Just not other metadata (last update date for example).

[JimmXinu]

No, there isn't.

The code looks for a raw numWords value, so even clearing it with replace_metadata doesn't work.

[booster-t]

Here's another data point regarding A03's 525 error. Trying to update this fic:

https://archiveofourown.org/works/61...ters/157176058

Calibre has 27 chapters and the index and chapters show chapter numbers. If I adjust the page on A03 to show Entire Work, I can download an epub with 28 chapters. However, chapter numbers are missing from the TOC and the file. Not sure if this is relevant but it's something ...

What a pain right now.

Stephen

[ThreeLizards]

Hi Jimmy, can I have your help? I looked at this https://github.com/JimmXinu/FanFicFa...Calibre-Plugin and it's not really helping me figure out my problem.

So, when I try to use fanficfare with the generate covers plugin, I get that screenshot below. Something about an error on line 2 at column 6: XML declaration only allowed at the start of the document. But, when I just use the generate covers plug in and press that, it generates the cover.

Can you please help me figure out why fanficfare is not working with the generate cover? I couldn't find anything in my personal.ini for it. and it's on the default cover.

Thank you!

[JimmXinu]

Without seeing the epub (or at least the story URL) and GC settings involved, no, I can't tell you anything about it.

[spicyfood]

Originally Posted by Clem2605

Hi!

I've begun tu use the browser cache/open_in_browser features with AO3 at least temporarily,because I was having way too many 525 errors without it, and I noticed something.

I use the option to download from Entire Works, and when opening the work in my browser, FFF opens both the navigation page of the work, and the work itself.

I'm assuming that FFF opens the navigation page to grab the links to all the chapters, but when downloading from entire works, this probably becomes redundant since FFF doesn't open those pages individually anymore.

Could it be possible to stop opening this page when the option to download from entire works is enabled?
This probably does more than I suspect, or maybe this isn't possible, but it would reduce by half the traffic to AO3 for each query if doable, so I thought it would be worth it to bring it up.

How do you do browser cache/open_in_browser? I'm a newb at this and I was trying to download/update in smaller amounts, but tonight I dried updating only 5 fics and they all got the 525 errors even though the plug-in is up to date.

[JimmXinu]

New Test Version Posted

2025-03-05
- BrowserCache Chrome Block: Treat entry missing headers same as not found. #1167 #1169

[ThreeLizards]

Okay, so for the cover thing, it's every single story I try to download.

Here is a URL for AO3 (I choose a G rated fic): https://archiveofourown.org/works/60...ters/155465734

Also I included a copy of the safe version of my personal.ini

Spoiler:

Safe Personal.Ini



## This is an example of what your personal configuration might look
## like.  Uncomment options by removing the '#' in front of them.

[defaults]
## [defaults] section applies to all formats and sites but may be
## overridden at several levels.  See
## https://github.com/JimmXinu/FanFicFare/wiki/INI-File for more
## details.

## Some sites also require the user to confirm they are adult for
## adult content.  Uncomment by removing '#' in front of is_adult.
is_adult:true

## Don't like the numbers at the start of chapter titles on some
## sites?  You can use strip_chapter_numbers to strip them off.  Just
## want to make them all look the same?  Strip them off, then add them
## back on with add_chapter_numbers.  Don't like the way it strips
## numbers or adds them back?  See chapter_title_strip_pattern and
## chapter_title_add_pattern in defaults.ini.
#strip_chapter_numbers:true
#add_chapter_numbers:true

always_overwrite: true

datePublished_format:%%B %%d, %%Y
dateUpdated_format:%%B %%d, %%Y
chapterslashtotal_label:Chapters

keep_in_order_freeformtags:true
keep_in_order_ships:true
keep_in_order_category:true
keep_in_order_characters:true
keep_in_order_fandoms:true
keep_in_order_genre:true
keep_in_order_ships:true

add_to_extra_valid_entries:,prefix_ships,ships_modified,category_modified,tags_modified,chapterslashtotal

include_in_prefix_ships:ships
include_in_ships_modified:ships
include_in_category_modified:category
include_in_tags_modified:freeformtags,characters,ships,warnings

replace_metadata: 

## Fandoms
 category_modified=>^Arcane: League of Legends \(Cartoon 2021\)$=>Arcane

# separate lines for ease of maintenance 
 category_modified=>^(Dragon Age).*$=>\1
 category_modified=>^(Mass Effect).*$=>\1
 category_modified=>^(Star Wars).*$=>\1
 category_modified=>^(Harry Potter).*$=>\1
 category_modified=>^(Buzzfeed Unsolved).*$=>\1

## Characters
 characters=>\(mentions of\) =>
 characters=> \([Ch]haracter\)=>
 characters=> ?\((cameo|briefly|if you squint|sort of)\)=>
 characters=> - Character=>
 characters=>^Other mentions$=>

 ships=> - Relationship=>

## Status
 status=>^In-Progress$=>WIP

## Removes images from summaries
 description=><img[^>]+>=>

## Removes fluff from categories
 category_modified=> \(?(- All Media Types|- Fandom|Movies|Movie [0-9]+|Comics|TV|Video Games?)\)?$=>

## Publisher
 site=>^archiveofourown.org$=>Archive of Our Own

## Ratings
 rating=>^General Audiences$=>G
 rating=>^Teen And Up Audiences$=>T
 rating=>^Mature$=>M
 rating=>^Explicit$=>E
 rating=>^Not Rated$=>U


## Standardize names for consistency
 ships,prefix_ships=>(OFC\/OFC)|(Original Female Character\/Original Female Character)=>Original Female Character(s)/Original Female Character(s)

exclude_metadata_pre:

## Removes "CharacterName" if Male or Female CharacterName is already present.
 characters=~^CharacterName&&characters=~^(Male|Female) CharacterName
## Exclude all entries that don't have / or &
 ships,prefix_ships!~(.*)(\/| &amp; )(.*)
## Exclude friendships in the ship tag
## remove this if you still want to see them
 ships,prefix_ships=~(.*) &amp; (.*)
## Exclude ship tags that are "implied/mentioned"
 prefix_ships=~^(.*)\(?(implied|mentioned)\)?
## Tags to be added removed
 additional_characters=~(Original|Various) Character\(?s?\)?
 freeformtags,genre==Other Additional Tags to Be Added


add_to_custom_columns_settings:
 ships_modified=>#pairings
 category_modified=>#fandom
 tags_modified=>#mytags
 kudos=>#ao3kudos_count
 hits=>#ao3hit_count
 bookmarks=>#ao3bookmark_count
 warnings=>#tags

# Add HTML to format the title page

titlepage_entries: series,rating,warnings,ao3categories,fandoms,ships,characters,freeformtags,language,collections,numWords,chapterslashtotal,description

titlepage_start:
 <?xml version="1.0" encoding="UTF-8"?>
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
 <html xmlns="http://www.w3.org/1999/xhtml">
 <head>
 <title>${title} by ${author}</title>
 <link href="stylesheet.css" type="text/css" charset="UTF-8" rel="stylesheet"/>
 <meta name="calibre-html-render-data"/>
 </head>
 <body>
 <div class="title">
 <a href="${storyUrl}">${title}</a> by ${authorHTML}
 </div>
 <dl class="titlepagelist">

titlepage_entry:
 <dt><b>${label}:</b></dt> <dd>${value}</dd>

## When using tables, make these span both columns.
wide_titlepage_entries: 

## ${id}, ${label}, ${value}
## Used with entries listed in wide_titlepage_entries, typically description, storyUrl, authorUrl
titlepage_wide_entry:
 <span><b>${label}:</b> ${value}</span>&nbsp;&nbsp;

## metadata
titlepage_end:
## <dl class="stats">
## </dl>
 </dl>
 <hr style="border: .75px solid #999; width: 90%%; margin: 10px auto;" />
 <br />
 <div style="text-align: left; font-weight: bold; margin-bottom: 5px;">Stats:</div>
 <div style="display: flex; justify-content: space-between; margin-bottom: 8px; width: 100%%;">
     <span style="flex: 1; text-align: left;">Published: ${datePublished}</span>
     <span style="flex: 1; text-align: left;">Updated: ${dateUpdated}</span>
     <span style="flex: 1; text-align: left;">Status: ${status}</span>
 </div>
 <div style="display: flex; justify-content: space-between; margin-bottom: 8px; width: 100%%;">
     <span style="flex: 1; text-align: left;">Kudos: ${kudos}</span>
     <span style="flex: 1; text-align: left;">Hits: ${hits}</span>
     <span style="flex: 1; text-align: left;">Bookmarks: ${bookmarks}</span>
   </div>
 </body>
 </html>


[epub]
## Include images from img tags in the body and summary of stories.
## Images will be converted to jpg for size if possible.  Images work
## in epub format only.  To get mobi or other format with images,
## download as epub and use Calibre to convert.
## true by default, uncomment and set false to not include images.
#include_images:true

## If set false, the summary will have all html stripped for safety.
## Both this and include_images must be true to get images in the
## summary.
## true by default, uncomment and set false to not keep summary html.
keep_summary_html:true

## If set true, and there isn't a specific cover image, the first
## image found in the story will be made the cover image.  If
## keep_summary_html is true, images in the summary will be before any
## in chapters.
## true by default, uncomment and set false to turn off
make_firstimage_cover:false


## Most common, I expect will be using this to save username/passwords
## for different sites.  Here are a few examples.  See defaults.ini
## for the full list.



output_css:
 body { background-color: #%(background_color)s;
        text-align: left;
        margin: 2%%;
        adobe-text-layout: optimizeSpeed; }
 pre { font-size: x-small; }
 sml { font-size: small; }
 h1 { text-align: center; }
 h2 { text-align: center; }
 h3 { text-align: center; }
 h4 { text-align: center; }
 h5 { text-align: center; }
 h6 { text-align: center; }
 .CI {
     text-align:center;
     margin-top:0px;
     margin-bottom:0px;
     padding:0px;
     }
 .center   {text-align: center;}
 .cover    {text-align: center;}
 .full     {width: 100%%; }
 .quarter  {width: 25%%; }
 .smcap    {font-variant: small-caps;}
 .u        {text-decoration: underline;}
 .bold     {font-weight: bold;}
 p { text-indent: 2em; }
 blockquote.userstuff {
    margin-bottom: 0.5em; /* Reduce spacing */
    padding-bottom: 0.5em;
    border-bottom: 1px solid #888;
 }

##/* But leave the summary's .userstuff alone (or style differently) */
 dd .userstuff {
    border-bottom: none;
    margin-bottom: 0;
    padding-bottom: 0;
 }


replace_xbr_with_hr:3
replace_hr:false
replace_br_with_p:false


[archiveofourown.org:epub]

add_to_output_css:
 dl.titlepagelist dd { margin: 0 0 1em 1em; }
 div.title {
  font-weight: bold;
  text-align: center;
 }

[archiveofourown.org]
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config.  In
## commandline version, this should go in your personal.ini, not
## defaults.ini.

use_workskin:false


freeformtags_label:Additional Tags
freefromtags_label:Additional Tags
ao3categories_label:Category
chapterslashtotal_label:Chapters
description_label: Summary

keep_in_order_ao3categories:true
keep_in_order_freeformtags:true


## This section will override anything in the system defaults or other
## sections here.

[overrides]
## default varies by site.  Set true here to force all sites to
## collect series.
#collect_series: true

So yeah, any help would be appreciated. It happens to every story. I don't know why.

[JimmXinu]

Originally Posted by ThreeLizards

Okay, so for the cover thing, it's every single story I try to download.
....

Yeah, that's not Generate Cover having a problem.

That's Calibre's "Set cover from ebook format" code called by having FFF's "Update Calibre Cover (from EPUB)" setting turned on, in combination with your custom titlepage_start setting.

You can see this happen manually if you Edit metadata in Calibre, choose the epub format and click the cover from format button:


Technically, the very start of an xhtml file has to be "<?xml ...?>". You have an extra new line.

# don't do this:
titlepage_start:
 <?xml version="1.0" encoding="UTF-8"?>
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
 ...

# do this:
titlepage_start:<?xml version="1.0" encoding="UTF-8"?>
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
 ...

The next question is: Why didn't the GC cover run and immediately hide that problem?

The answer to that is that the GC GUI selection uses FFF's site metadata entry--and you've changed it in replace_metadata.

There are certain metadata entries FFF uses internally that, technically can be changed by the user, but will break certain features. Especially, storyUrl, but also site, & status.

So: You can either not change site, use Default GUI GC instead of archiveofourown.org, or use generate_cover_settings to specify the GC setting to use.

FYI, publisher is a copy of site that gets put in Calibre's Publisher field. That, you can safely change.

From defaults.ini:

## The metadata entry 'site' predates Calibre integration, and was
## later labeled Publisher and used to fill Calibre's Publisher field.
## Because users keep expecting it to be 'publisher' instead,
## 'publisher' is now a copy of 'site' and the Calibre plugin now uses
## 'publisher' instead.  Default titlepage_entries are also changed.
include_in_publisher:site
publisher_label:Publisher