[GUI Plugin] FanFicFare

[ownedbycats]

I assume DisturbedDragon is not the owner of the router.

[gthesob]

Originally Posted by DisturbedDragon

Turns out my family installed NordVPN on my router without informing me. It's what is causing my logins to sv/sb to fail. How would you suggest I bypass this issue? Is there an exception I have to make somewhere?
What process name do I need to allow?

Assuming it has the ability, whitelist the URL.

[DisturbedDragon]

Fixed it by installing Nord on the desktop, while removing it from the router, so my laptop can update while their browsing is still secure. I'm now just annoyed they didn't think to tell me about this beforehand

[culytera]

Back again with another question, is it possible to format the date depending on whether the date published and date updated are the same or not? For example a fic from Ao3 has a published date of January 1, 2022 and an updated date of January 2, 2022, the final date string should be January 1-2, 2022. If it's a different month, January 1-February 2, 2022. If a different year, January 1, 2021-February 2, 2022.

So far this is what I've done:

add_to_extra_valid_entries:,finalDate,publishedMonth,updatedMonth,publishedDay,updatedDay,publishedYear,updatedYear

include_in_publishedMonth:datePublished
include_in_updatedMonth:dateUpdated
include_in_publishedDay:datePublished
include_in_updatedDay:dateUpdated
include_in_publishedYear:datePublished
include_in_updatedYear:dateUpdated

publishedMonth_format:%%B
updatedMonth_format:%%B
publishedDay_format:%%d
updatedDay_format:%%d
publishedYear_format:%%Y
updatedYear_format:%%Y

But changing the format didn't work, I tried to manually change them via regex but that didn't work either.

add_to_replace_metadata:
 publishedMonth,updatedMonth=>(January|February|March|April|May|June|July|August|September|October|November|December)(\s+\d{1,2})(,\s+)(\d{4})=>\1
 publishedDay,updatedDay=>(January|February|March|April|May|June|July|August|September|October|November|December)(\s+\d{1,2})(,\s+)(\d{4})=>\2
 publishedYear,updatedYear=>(January|February|March|April|May|June|July|August|September|October|November|December)(\s+\d{1,2})(,\s+)(\d{4})=>4

Then I was thinking of comparing each published/updated month/day/year to form the final date, but I don't know if replace_metadata can take the value from other variables. Don't know why I can't just be satisfied with my current layout but here we are lol

[JimmXinu]

Originally Posted by culytera

Back again with another question, is it possible to format the date depending on whether the date published and date updated are the same or not?
...
Then I was thinking of comparing each published/updated month/day/year to form the final date, but I don't know if replace_metadata can take the value from other variables. Don't know why I can't just be satisfied with my current layout but here we are lol

(date)_format only works on the three standard date entries: dateCreated, datePublished, and dateUpdated. The include_in_X mechanism make a list of strings, so it's not practical to change that.

And each replace_metadata line only works on one value at a time without knowledge of any other entry.

But I think it's possible to do what you want by using the _LIST feature of replace_metadata, this seems to work for me:

datePublished_format:%%B %%d, %%Y
dateUpdated_format:%%B %%d, %%Y
add_to_extra_valid_entries:,dateRange
include_in_dateRange:datePublished,dateUpdated
keep_in_order_dateRange:true

add_to_replace_metadata:
# same year      1          2            3        4          5
 dateRange_LIST=>([^\s]+)\s+(\d{1,2}),\s+(\d{4}), ([^\s]+)\s+(\d{1,2}),\s+\3=>\1 \2-\4 \5, \3
# same month     1          2              3            4
 dateRange_LIST=>([^\s]+)\s+(\d{1,2})-\1\s+(\d{1,2}),\s+(\d{4})=>\1 \2-\3, \4
# don't need regexp for date day, handled by de-dup.

1. Explicitly set date formats--some sites also include times, those will be lost unfortunately
2. Make new entry dateRange
3. Populate dateRange with datePublished, dateUpdated
4. Keep dateRange in order (otherwise alphabetized, not date ordered)
5. Replace on dateRange_LIST (the whole list) looking for the same year -- note \3 matching the second year
6. Replace on dateRange_LIST (the whole list) looking for the same month ONLY on strings that already had the same year -- note \1 matching the second month and - instead of ,\s+ between them.
7. datePublished == dateUpdated and stories without dateUpdated will show up in dateRange as just one date.

[culytera]

Originally Posted by JimmXinu

But I think it's possible to do what you want by using the _LIST feature of replace_metadata
[...]

Ah so it wasn't as complicated as I thought, thank you. Though it only seems to be working for dates that are the exact same. When I check what dateRange is for the ones that didn't work the string is something like "April 11, 2021, September 15, 2021" or "March 15, 2019, March 16, 2019" so the two regexp's should've worked accordingly.

[JimmXinu]

Originally Posted by culytera

Ah so it wasn't as complicated as I thought, thank you. Though it only seems to be working for dates that are the exact same. When I check what dateRange is for the ones that didn't work the string is something like "April 11, 2021, September 15, 2021" or "March 15, 2019, March 16, 2019" so the two regexp's should've worked accordingly.

The replacements are working for me when I test.

I'd have to see the story URLs and your personal.ini(there's a button for a safe version) to comment on why those particular strings didn't.

[culytera]

Originally Posted by JimmXinu

The replacements are working for me when I test.

I'd have to see the story URLs and your personal.ini(there's a button for a safe version) to comment on why those particular strings didn't.

I restructured my settings for ao3 and it works now. Thanks again for all your help.

It might have had something to do with the order of my settings. I also added another one for completely different dates so the structure is "published date-updated date" instead of "published date, updated date". Sharing my .ini anyway for posterity.

Spoiler:

Some test URLs for different dates.

https://archiveofourown.org/works/19072861
https://archiveofourown.org/works/9150928
https://archiveofourown.org/works/18119894
https://archiveofourown.org/works/18207158

My personal.ini. I suppose all I have left to do is making the metadata stuff more efficient, but I'm quite satisfied with the result now.

## This is an example of what your personal configuration might look
## like.  Uncomment options by removing the '#' in front of them.


[test1.com]
# URL: http://test1.com?sid=12345


[epub]
## include images from img tags in the body and summary of stories.
## Images will be converted to jpg for size if possible.  Images work
## in epub format only.  To get mobi or other format with images,
## download as epub and use Calibre to convert.
include_images:true


## If not set, the summary will have all html stripped for safety.
## Both this and include_images must be true to get images in the
## summary.
#keep_summary_html:false


## If set, the first image found will be made the cover image.  If
## keep_summary_html is true, any images in summary will be before any
## in chapters.
make_firstimage_cover:false


jpg_quality:100
convert_images_to:png


## Remove transparency and fill with background_color if true.
remove_transparency: false


## This switch prevents FanFicFare from doing any processing on the images.
## Usually they would be converted to jpg, resized and optionally made
## grayscale.
no_image_processing: true


## This section will override anything in the system defaults or other
## sections here.
[overrides]
## Don't like the numbers at the start of chapter titles on some
## sites?  You can use strip_chapter_numbers to strip them off.  Just
## want to make them all look the same?  Strip them off, then add them
## back on with add_chapter_numbers.  Don't like the way it strips
## numbers or adds them back?  See chapter_title_strip_pattern and
## chapter_title_add_pattern.
strip_chapter_numbers:true
add_chapter_numbers:true


## Add this to genre if there's more than one category.
#add_genre_when_multi_category: Crossover


## Some sites/authors/stories use several br tags for scene/section
## breaks.  When set replace_xbr_with_hr:X will cause FFF to search
## for X or more consecutive br tags and replace them with br br hr br.
replace_xbr_with_hr:12


## Some readers don't show horizontal rule (<hr />) tags correctly.
## This replaces them all with a centered '* * *'.  (Note centering
## doesn't work on some devices either.)
replace_hr:false


## Some sites/authors/stories use br tags instead of p tags for
## paragraphs.  This feature uses some heuristics to find and replace
## br paragraphs with p tags while preserving scene breaks.
replace_br_with_p:false


## By default, empty tags are removed as part of cleaning up the
## source HTML.  However, a few tags should be kept even if empty.
## (Whitespace only, including &nbsp; is considered empty.)  This
## setting can adjust which tags are kept.
keep_empty_tags:p,br
#p,td,th


[archiveofourown.org]
include_in_genre:
## freeformtags, ao3categories
is_adult:true
use_view_full_work:true


## Date formats used by FanFicFare.  Published and Update don't have time.
## See http://docs.python.org/library/datetime.html#strftime-strptime-behavior
## Note that ini format requires % to be escaped as %%.
datePublished_format:%%B %%d, %%Y
dateUpdated_format:%%B %%d, %%Y


add_to_extra_valid_entries:,prefix_ships,prefix_category,prefix_rating,prefix_status,keep_ships,dateRange

include_in_prefix_ships:ships
include_in_keep_ships:ships
include_in_prefix_category:category
include_in_prefix_rating:rating
include_in_prefix_status:status
include_in_dateRange:datePublished, dateUpdated


category_label:Fandom
characters_label:Additional Characters
freeformtags_label:Additional Tags
freefromtags_label:Additional Tags
ao3categories_label:Category
chapterslashtotal_label:Chapters


exclude_metadata_pre:
 freeformtags,genre==Other Additional Tags to Be Added
 category==supercorp - Fandom
 warnings==Creator Chose Not To Use Archive Warnings
 warnings==No Archive Warnings Apply
 ships!~(.*)&amp;(.*)|(.*)\/(.*)
## Removes the character in Characters metadata if they are already included in Relationships metadata
 characters=~(Kara Danvers)&&ships=~(Kara Danvers)
 characters=~(Lena Luthor)&&ships=~(Lena Luthor)
 characters=~(Andrea Rojas)&&ships=~(Andrea Rojas)
 characters=~(Carol Danvers)&&ships=~(Carol Danvers)
 characters=~(Maria Rambeau)&&ships=~(Maria Rambeau)
 characters=~Raya&&ships=~Raya
 characters=~Namaari&&ships=~Namaari
 characters=~Regina Mills&&ships=~Regina Mills
 characters=~Emma Swan&&ships=~Emma Swan
 characters=~Korra&&ships=~Korra
 characters=~Asami Sato&&ships=~Asami Sato
 characters=~Samantha "Sam" Arias&&ships=~Samantha "Sam" Arias
 characters=~Alex Danvers&&ships=~Alex Danvers
 characters=~Kelly Olsen&&ships=~Kelly Olsen
 characters=~Maggie Sawyer&&ships=~Maggie Sawyer
 characters=~Querl Dox&&ships=~Querl Dox
 characters=~Nia Nal&&ships=~Nia Nal
 characters=~Russell Rogers&&ships=~Russell Rogers
 characters=~Miranda Priestly&&ships=~Miranda Priestly
 characters=~Andrea Sachs&&ships=~Andrea Sachs
 prefix_ships=~Alex Danvers[/\s&].*
 ships,prefix_ships==Lena Luthor/Andrea Rojas
 prefix_ships==Mulan/Red Riding Hood | Ruby
 prefix_ships==Oliver Queen/Felicity Smoak
 prefix_ships==Querl Dox/Nia Nal
 prefix_ships==Russell Rogers/Andrea Rojas
 prefix_ships==Samantha "Sam" Arias/Alex Danvers
 ships,prefix_ships=="Hyde" Evil Queen/Emma Swan
 ships,prefix_ships==Evil Queen | Regina Mills/"Hyde" Evil Queen
 ships==Female Altmer/Female Dunmer
 ships==Female Vestige/OC
 prefix_category=~Doctor Who
 category,prefix_category==Elder Scrolls Online
 prefix_category=~Good Omens
 category,prefix_category=~Marvel Cinematic Universe
 prefix_category=~Merlin
 prefix_category=~Schitt's Creek
 prefix_category=~The Parent Trap
 ships,prefix_ships=~(.*) &amp; (.*)&&category=~Once Upon a Time|Captain Marvel


replace_metadata:
 category=> (- All Media Types|- Fandom|\(Movies\)|\(Movie [0-9]+\)|\(Comics\)|\(TV\))$=>
 rating=> Audiences=>
 characters=>\(mentions of\) =>
 characters=> \([Ch]haracter\)=>
 characters=> ?\((cameo|briefly|if you squint|sort of)\)=>
 characters=> - Character=>
 characters=>^Other mentions$=>
 ships=>(.*)[/&](.*) \(?[Ff]riendship(?: only)\)?=>\1 & \2
 ships=> - Relationship=>
 ships,category= - Fandom$=>
 ships=>\((mentioned|implied)(?!\))=>(\1)
 ships=>^(.* - Relationship)$=>
 ships=>^(Eventual (.*)/(.*))$=>
 ships=>^(Lena Luthor/Kara Danvers)$=>Kara Danvers/Lena Luthor
 ships=>^(Lena Luthor/Kara Zor-El)$=>Kara Danvers/Lena Luthor
 ships,characters=> \(Disney\)$=>
 characters=>( \(Supergirl TV 2015\))$=>
 characters=>( \(Once Upon a Time\))$=>
 characters=>( \(Avatar\))$=>
 characters=>^(Other\(s\))$=>
 prefix_ships=>^(.*)=>[p] \1
 prefix_category=>^(.*)=>[c] \1
 prefix_rating=>^(.*)=>[r] \1
 prefix_status=>^(.*)=>[s] \1
 prefix_ships=>^(\[p\] ).* (\()?(implied|mentioned)(\))?=>


## Removes the platonic ship tag if ship is already mentioned in a romantic context
exclude_metadata_post:
 ships==Kara Danvers &amp; Lena Luthor&&keep_ships==Kara Danvers/Lena Luthor
 ships=~(Evil Queen \| Regina Mills &amp; Emma Swan)&&keep_ships=~(Evil Queen \| Regina Mills/Emma Swan)
 ships=~Namaari &amp; Raya&&keep_ships=~Namaari/Raya


keep_in_order_ao3categories:true
keep_in_order_category:true
keep_in_order_characters:true
keep_in_order_dateRange:true
keep_in_order_fandoms:true
keep_in_order_freeformtags:true
keep_in_order_genre:true
keep_in_order_ships:true


add_to_replace_metadata:
# diff dates     1pmonth    2pday        3pyear   4umonth    5uday        6uyear
 dateRange_LIST=>([^\s]+)\s+(\d{1,2}),\s+(\d{4}), ([^\s]+)\s+(\d{1,2}),\s+(\d{4})=>\1 \2, \3-\4 \5, \6
# same year      1          2            3        4          5
 dateRange_LIST=>([^\s]+)\s+(\d{1,2}),\s+(\d{4})-([^\s]+)\s+(\d{1,2}),\s+\3=>\1 \2-\4 \5, \3
# same month     1          2              3            4
 dateRange_LIST=>([^\s]+)\s+(\d{1,2})-\1\s+(\d{1,2}),\s+(\d{4})=>\1 \2-\3, \4
# don't need regexp for date day, handled by de-dup.


add_to_custom_columns_settings:
 category=>#ao3_category
 chapterslashtotal=>#ao3_chapters
 characters=>#ao3_characters
 dateRange=>#ao3_date
 dateUpdated=>#ao3_dateupdated
 freeformtags=>#ao3_additional_tags
 numWords=>#ao3_words
 rating=>#ao3_rating
 ships=>#ao3_ships


## ships, category, rating, status with the added prefixes in Calibre's tags
include_subject_tags:prefix_ships,prefix_category,prefix_rating,prefix_status


## extra tags (comma separated) to include, primarily for epub.
extratags:


add_to_output_css:
 body {
  font-family: "Roboto", sans-serif;
  text-align: left;
 }
 body.fff_titlepage {
  background-color: #980101;
  color: white;
 }
 .stats { margin-top: 0.5em; }
 .hr {
   display: block; height: 1px;
   border: 0; border-top: 1px solid;
   margin: 0.6em auto; padding: 0; width: 75%%;
  }
 .fff_titlepage .title h1,
 .fff_titlepage .title h2,
 .fff_titlepage .title a {
  margin: 0;
  color: inherit;
 }
 .fff_titlepage .title a,
 .fff_titlepage .tags a {
  text-decoration: none;
  color: inherit;
 }
## .tag-summary-content a { text-decoration: none; color: white; }
 .tags {
  display: block;
  padding: 0;
  margin: 0.175em 0;
 }
 dl .inline dd {
   display: inline;
   margin: 0;
 }
 dl .inline dd:after{
   display: block;
   content: '';
 }
 dl .inline dt{
   display: inline-block;
 }
 dd { margin-left: 0.75em; }
 dl, dt { margin: 0; }


## Items to include in the title page
## Empty metadata entries will *not* appear, unless .SHOW_EMPTY is
## appended.  Eg: titlepage_entries: ..., characters.SHOW_EMPTY,
## ships,... will always display Characters, but only show
## Relationships if there are any.  You can include extra text or HTML
## that will be included as-is in the title page. Eg:
## titlepage_entries: ...,<br />,summary,<br />,...
## All current formats already include title and author.
titlepage_entries:<div class="inline">,seriesHTML,category,rating,warnings,</div>,ships,characters,freeformtags


## metadata ${title}, etc.  Make sure to keep at least one space
## at the start of each line and to escape % to %%.


titlepage_start:<?xml version="1.0" encoding="UTF-8"?>
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
 <html xmlns="http://www.w3.org/1999/xhtml">
 <head>
 <title>${title} by ${author}</title>
 <link href="stylesheet.css" type="text/css" charset="UTF-8" rel="stylesheet"/>
 <meta name="calibre-html-render-data" content='{"size": "A2", "margins": [0, 0, 0, 0]}'/>
 </head>
 <body class="fff_titlepage" style="font-size: 2.7vmax;">
 <div class="title">
 <h1><a href="${storyUrl}">${title}</a> by ${authorHTML}</h1>
 </div>
 <hr/>
 <dl class="tags">


## ${id}, ${label}, ${value}
titlepage_entry:
 <dt><b>${label}:</b></dt> <dd>${value}</dd>


## ${id}, ${label}, ${value}
## Used with entries listed in wide_titlepage_entries, typically description, storyUrl, authorUrl
titlepage_wide_entry:


## When using tables, make these span both columns.
wide_titlepage_entries:


## ${id}, ${value}
## For example, if you've set description_label:
## (that is, nothing)
titlepage_no_title_entry:


## metadata
titlepage_end:
 <dl class="stats">
 <dt><b>Words & Chapters:</b> ${numWords} &bull; ${chapterslashtotal}</dt>
 <dt><b>Published & Updated:</b> ${dateRange}</dt>
 </dl>
 </dl>
 </body>
 </html>

Looks nice now when using the Generate Cover plugin.

[firefoxxy]

I have a regex question. My ini the following lines:

## Full name or additional info for relationships
## 1. Make these two word names one word before the 'first part only' line.
 ships_CHARS=>[pP]ast (.*)=>past_\1
## strip each ships char down to first part only
 ships_CHARS=>^([^ ]+).*$=>\1
## 2. Change the two words names back into separate words
 ships_CHARS=>past_(.*)=>past \1

(This is just the relevant lines. If the full ini is necessary, I'll post it.)

For example, in this story https://archiveofourown.org/works/28197405, the regex results in "Claude/past Sylvain", which is expected. What I would like to get is something like "past Claude/Sylvain".

I'm not sure how to do that because FFF sorts the ships alphabetically, but in this instance I don't want it to sort the ship. I want "past" to be the first part of the whole ship. Could someone help me?

[JimmXinu]

Originally Posted by firefoxxy

I have a regex question. My ini the following lines:
...
I'm not sure how to do that because FFF sorts the ships alphabetically, but in this instance I don't want it to sort the ship. I want "past" to be the first part of the whole ship. Could someone help me?

FFF only sorts the ships alphabetically when you turn on sort_ships:true.

ships_CHARS specifically operates on the individual characters, those replace lines can't see the whole ships entry value they come from.

What seems to work for me is to temporarily split 'past' out as it's own 'person' (\, operator), force it to be at the front, then replace it later:

sort_ships:true

add_to_replace_metadata:
# Split leading 'past' to own (temp) entry, 00 to put at
# front(sort_ships:true assumed)
 ships_CHARS=>^[pP]ast (.*)=>00past\,\1
## strip each ships char down to first part only
 ships_CHARS=>^([^ ]+).*$=>\1
## Change 00past/ back to "Past "
 ships=>^00past\s*(/|&)\s*=>Past\s

Note that in this case, both 'Past Z/A' and 'Z/Past A' will become 'Past A/Z'.

[estherflails]

Hello! I modified the ao3 adapter a bit to get the inspired works metadata and it works, but only for single chaptered stories, since this information only shows up in the last chapter.

But since there is a method for FFF to use the full_work view to download the entire work, I was wondering if it was possible to use this to get the metadata from the end as well. I'm not a programmer so I don't know how complicated this would be. It's not a problem if it's not possible, but I thought I'd ask. Thank you!

example link: https://archiveofourown.org/works/29...pters/71633358

inspired = metasoup.find('div',{'id':'children'})
if inspired:
    self.story.setMetadata('inspired',stripHTML(inspired.text))

[firefoxxy]

Originally Posted by JimmXinu

What seems to work for me is to temporarily split 'past' out as it's own 'person' (\, operator), force it to be at the front, then replace it later

Thank you very much, Jimm! That worked like a charm.

[JimmXinu]

Originally Posted by estherflails

Hello! I modified the ao3 adapter a bit to get the inspired works metadata and it works, but only for single chaptered stories, since this information only shows up in the last chapter.
...

I didn't remember it, but this was brought up once before.

Because it's at the end of the last chapter, it's included as a chapter note element and controlled by the exclude_notes setting. There are other notes that are only included at the end of the 'final' chapter--which moves when a story updates.

FFF operates on a paradigm of getting metadata separately from chapter data. Metadata is collected first, then used to make decisions about chapters to fetch and to allow for metadata update with less traffic. FFF does have a AO3 only feature use_view_full_work (on by default), but that's not invoked until chapter fetching.

To reliably include AO3 'Inspired by' links in the metadata would require fetching the last chapter text in extractChapterUrlsAndMetadata() to look for the 'inspired by' links. Ideally, it would then use the make_linkhtml_entries feature (and keep_in_order_) to make proper HTML links rather than just a strings. The author links could not currently preserved.

IMO, it's rare enough to not be worth the extra network hit. But if you really want it, I'd accept it as a default-off optional feature.

[estherflails]

Originally Posted by JimmXinu

I didn't remember it, but this was brought up once before.

Thank you, but I figured it would be good enough if I could have it set metadata if a story has works inspired by it, which is noted in the first chapter. I got it to work by using the "jump" class but unfortunately that also exists when the story has endnotes, not just inspired works, and I can't figure out how to get around that.

inspired = metasoup.find('div',{'id':'children'})
seetheend = metasoup.find('p',{'class':'jump'})
if inspired:
    self.story.setMetadata('inspired',stripHTML(inspired.text))
elif seetheend:
    self.story.setMetadata('seetheend',stripHTML(seetheend.text))

multi-chaptered work with inspired: https://archiveofourown.org/works/29...pters/71633358
single chaptered work with inspired: https://archiveofourown.org/works/234459
single chaptered work with endnotes: https://archiveofourown.org/works/3968443

Also, does make_linkhtml_entries only work for the titlepage? Is there a way to preserve the links in long text columns, or would that not work since the links are only partial?

[JimmXinu]

Originally Posted by estherflails

Thank you, but I figured it would be good enough if I could have it set metadata if a story has works inspired by it, which is noted in the first chapter. I got it to work by using the "jump" class but unfortunately that also exists when the story has endnotes, not just inspired works, and I can't figure out how to get around that.

You'd likely be better served by find('a',text='more notes') and text='other works inspired by this one'. Do find('p',{'class':'jump'}) first if you want to be extra sure--I can see 'more notes' being an author-inserted link in the story text.

BTW, doing both .text and stripHTML() is redundant. I prefer stripHTML() because it does some other stuff and definitely gets all sub tags.

I would suggest that if you're just making them general links you change the text anyway. "End Notes" and "Works inspired by this one", for example, instead of just keep the lower case as-is.

Originally Posted by estherflails

Also, does make_linkhtml_entries only work for the titlepage? Is there a way to preserve the links in long text columns, or would that not work since the links are only partial?

I haven't tested it recently, but as I recall, it worked to populate a long text(HTML) custom column using yourentryHTML in custom_columns_settings.

As for partial links, it's the adapter file's responsibility to finish out the URL before setting it into metadata. Look in adapter_archiveofourownorg.py for all the places it does 'https://' + self.getSiteDomain() + alink['href'] and such.