[GUI Plugin] FanFicFare

[slimcat]

Originally Posted by JimmXinu

Please note that if you remove those files, FFF will not be able to update the anthology epub because you will have removed the files EpubMerge needs to 'UnMerge' it.

EDIT: As a general note, changing the internals of an FFF generated epub runs the risk of breaking FFF's ability to update it. Changing/fixing chapter text is usually safe. Changing the structure is often not.

Sorry, yes I should have noted this is only for completed anthologies.

[eb21145]

I'm having an issue with the order of operations with exclude_metadata_pre, replace_metadata, and exclude_metadata_post. I'm trying to standardized and remove duplicative tags so I've got things set up so my code looks like this:

## standardize tags

replace_metadata:

 freeformtags,genre=>^(.*) and Angst=>Angst and \1
 freeformtags,genre=>^.* Angst=>Angst

## exclude duplicative angst tags

exclude_metadata_post:
 freeformtags,genre=~^Angst&&freeformtags,genre==Angst [Aa]nd .*
 freeformtags,genre=~^Angst&&freeformtags,genre==Angst [Ww]ith .*

However this pattern results in anything that had a ".* Angst" tag (e.g.: Light Angst) from being completely excluded from my freeformtags column, even if the story doesn't have an "Angst and X" or "Angst with X" pattern, rather than showing up as just "Angst" -- e.g., a fic with the tags "Light Angst, Case Fic" becomes tagged with just "Case Fic" rather than getting the tags "Angst, Case Fic". When I move the exclusion rule to exclude_metadata_pre, everything works as I would expect it to but it does not capture any tags there were modified by the replace_metadata -- E.g., if a fic had the tags "Angst with a Happy Ending" and "Light Angst", the freeformtags column would become "Angst, Angst with a Happy Ending" rather than just "Angst with a Happy Ending".

Am I misunderstanding how exclude_metadata_post works?

[JimmXinu]

Originally Posted by eb21145

I'm having an issue with the order of operations with exclude_metadata_pre, replace_metadata, and exclude_metadata_post. I'm trying to standardized and remove duplicative tags so I've got things set up so my code looks like this:

...

However this pattern results in anything that had a ".* Angst" tag (e.g.: Light Angst) from being completely excluded from my freeformtags column, even if the story doesn't have an "Angst and X" or "Angst with X" pattern, rather than showing up as just "Angst" -- e.g., a fic with the tags "Light Angst, Case Fic" becomes tagged with just "Case Fic" rather than getting the tags "Angst, Case Fic". When I move the exclusion rule to exclude_metadata_pre, everything works as I would expect it to but it does not capture any tags there were modified by the replace_metadata -- E.g., if a fic had the tags "Angst with a Happy Ending" and "Light Angst", the freeformtags column would become "Angst, Angst with a Happy Ending" rather than just "Angst with a Happy Ending".

Am I misunderstanding how exclude_metadata_post works?

• Conditionals (the && part) only use one key.
  &&freeformtags,genre==Angst [Aa]nd .* is only going to use the first key, freeformtags.
  
• Any line is ignored when the conditional clause key causes recursion.
  To calculate the value of genre, all of the replace_metadata and exclude_metadata_post lines for genre have to be called--if genre is called during that, it would recurse forever.
  
  You have to either use a different metadata entry, or make a copy that isn't processed See wiki.
  
• Your conditionals use ==, which matches exact value, while apparently trying to use regular expression Angst [Aa]nd .*.
  
• The _pre and _post refer to before or after replace_metadata is done.

If you post example story URLs, I will try to help with specific problems. But I honestly consider futile to try normalizing AO3 tags. I'm going to put the full rant in another post.

[JimmXinu]

Here's my AO3 tags rant in full, since it's been a while. We've been talking about this for over a decade. (I wonder what I was reading at the time that I came out with "...I choose not to plow the sea.")

Most of the arguments apply to any site that allows free form metadata on stories.

AO3 Tags Cannot be Normalized Automatically

AO3 tagging, because it allows every author to enter anything they want into any story field, is impossible to automatically normalize.

Too Many Options

Any popular tag has tons of alternates. Here's just one example: Naruto, according to AO3 itself, has 300 "Tags with the same meaning".

Misspellings. Just all the misspellings. Just two: Naruro NRUTO

Flat out wrong tags. Tag 1234. Some author made a tag '1234' and now it means Naruto? Seriously?

light saber duel is not a Fandom -- it's not even a useful tag because it's been made a synonym; its 'Works' button gets you all of 'Star Wars - All Media Types'. So good luck finding that story.

Inconsistent Conventions

For example, ships. We assume '/' means romantic ship and ' & ' platonic. Except we've seen AO3 ships with both & and / in them. With and without the spaces around, ie 'A / B' and 'A&B'. And order, eg, A/B vs B/A, multiplied out by however many in the polycule, etc.

Humor vs Humour vs "humor" vs Comedy, etc, etc. Multiply by every genre.

Too Many Tags

We've seen AO3 stories with 100s of tags, including ones where the author wrote the whole story description in the tags. And again in the description field.

Too Much Overlap

Thor. Is that Thor from Norse methology, Supernatural, MCU, Marvel Comics? Should they be differentiated?

If you write a rule in FFF to handle Thor, you also have to handle "Thor" vs "thor" vs "sexy!Thor" while also not affecting "Thorn" or "Arthor". Multiply by 100s of characters. And fandoms. And ships.

AO3 Wrangled / Canonical Tags

AO3 has an entire team of people to 'wrangle' tags, and it's complex enough they have to have a huge set of guidelines on the subject.

FFF fetching the page for every tag to get a 'canonical' version is not feasible. There's just too many tags on too many stories.

If AO3 had an option or page that gave the wrangled/canonical tags for a story, that I would look at very seriously.

Is It Really Impossible?

Really, really? If you are willing to do the work and discard everything that doesn't fit your scheme, you can get normalized metadata.

But my experience has been that the people asking to normalize metadata are the ones least willing to discard data.

Or maybe I just don't hear from people who've solved it to their satisfaction.

This Metadata Management for Fanfiction thread might help (but I don't follow it myself).

In Conclusion / What I Will Help With

I'm not interested in trying to solve metadata normalization in general.

It's been my policy for a very long time that I will help with specific metadata processing problems. And by specific, I mean you ideally have:

• The personal.ini code you're trying to use,
• A clear description of what you're trying to get,
• Example story URLs that do not give you what you want (plus example URLs that do work), and;
• Clear descriptions of what's wrong with the output from those examples.

[DNSB]

Originally Posted by JimmXinu

Here's my AO3 tags rant in full, since it's been a while. We've been talking about this for over a decade. (I wonder what I was reading at the time that I came out with "...I choose not to plow the sea.")

I seem to remember reading "to plow the sea" in an biography of Simón Bolívar as a quote about his belief in later life that his actions in freeing South America from Spanish rule had ultimately been futile.

[JimmXinu]

Originally Posted by DNSB

I seem to remember reading "to plow the sea" in an biography of Simón Bolívar as a quote about his belief in later life that his actions in freeing South America from Spanish rule had ultimately been futile.

I bet I was listening to Mike Duncan's Revolutions podcast covering South America at the time.

[ownedbycats]

Originally Posted by JimmXinu

Here's my AO3 tags rant in full, since it's been a while. We've been talking about this for over a decade. (I wonder what I was reading at the time that I came out with "...I choose not to plow the sea.")

Most of the arguments apply to any site that allows free form metadata on stories.

AO3 Tags Cannot be Normalized Automatically

AO3 tagging, because it allows every author to enter anything they want into any story field, is impossible to automatically normalize.

Too Many Options

Any popular tag has tons of alternates. Here's just one example: Naruto, according to AO3 itself, has 300 "Tags with the same meaning".

Misspellings. Just all the misspellings. Just two: Naruro NRUTO

Flat out wrong tags. Tag 1234. Some author made a tag '1234' and now it means Naruto? Seriously?

light saber duel is not a Fandom -- it's not even a useful tag because it's been made a synonym; its 'Works' button gets you all of 'Star Wars - All Media Types'. So good luck finding that story.

Inconsistent Conventions

For example, ships. We assume '/' means romantic ship and ' & ' platonic. Except we've seen AO3 ships with both & and / in them. With and without the spaces around, ie 'A / B' and 'A&B'. And order, eg, A/B vs B/A, multiplied out by however many in the polycule, etc.

Humor vs Humour vs "humor" vs Comedy, etc, etc. Multiply by every genre.

Too Many Tags

We've seen AO3 stories with 100s of tags, including ones where the author wrote the whole story description in the tags. And again in the description field.

Too Much Overlap

Thor. Is that Thor from Norse methology, Supernatural, MCU, Marvel Comics? Should they be differentiated?

If you write a rule in FFF to handle Thor, you also have to handle "Thor" vs "thor" vs "sexy!Thor" while also not affecting "Thorn" or "Arthor". Multiply by 100s of characters. And fandoms. And ships.

AO3 Wrangled / Canonical Tags

AO3 has an entire team of people to 'wrangle' tags, and it's complex enough they have to have a huge set of guidelines on the subject.

FFF fetching the page for every tag to get a 'canonical' version is not feasible. There's just too many tags on too many stories.

If AO3 had an option or page that gave the wrangled/canonical tags for a story, that I would look at very seriously.

Is It Really Impossible?

Really, really? If you are willing to do the work and discard everything that doesn't fit your scheme, you can get normalized metadata.

But my experience has been that the people asking to normalize metadata are the ones least willing to discard data.

Or maybe I just don't hear from people who've solved it to their satisfaction.

This Metadata Management for Fanfiction thread might help (but I don't follow it myself).

In Conclusion / What I Will Help With

I'm not interested in trying to solve metadata normalization in general.

It's been my policy for a very long time that I will help with specific metadata processing problems. And by specific, I mean you ideally have:

• The personal.ini code you're trying to use,
• A clear description of what you're trying to get,
• Example story URLs that do not give you what you want (plus example URLs that do work), and;
• Clear descriptions of what's wrong with the output from those examples.

Should make this a wiki page for easy linking.

[eb21145]

Originally Posted by JimmXinu

[LIST]
If you post example story URLs, I will try to help with specific problems. But I honestly consider futile to try normalizing AO3 tags. I'm going to put the full rant in another post.

Ahh, thanks for the explanation. I'm still fumbling my way through all of this.

Well it's annoying to have repetitive entries but not the end of the world.

[Jade Aislin]

Originally Posted by JimmXinu

Is It Really Impossible?

Really, really? If you are willing to do the work and discard everything that doesn't fit your scheme, you can get normalized metadata.

But my experience has been that the people asking to normalize metadata are the ones least willing to discard data.

Or maybe I just don't hear from people who've solved it to their satisfaction.

Personally, I'm just trying to normalize as much as I can for re-occuring tags. Then I'll manually delete stuff I don't want.

I don't help my case when I decide to totally rework how I handle pairings. . . . . I do appreciate any help you give, even if I'm thinking of a new way to get what I want.

[sparrowbby]

HTTP Error in FFF '403 Client Error: Forbidden for url: https://archiveofourown.gay/users/login'(403)


Removed the work URL because I don't want to publicly blast what I'm trying to download at random, haha.

This is what's in my personal.ini (redacted the username and PW, of course), but am I missing/misformatting something? It was fine last night, but I did put everything in plugin-defaults.ini since that's the only spot I could find the ao3 username/password combo...

## This is an example of what your personal configuration might look
## like. Uncomment options by removing the '#' in front of them.

[defaults]
## [defaults] section applies to all formats and sites but may be
## overridden at several levels. See
## https://github.com/JimmXinu/FanFicFare/wiki/INI-File for more
## details.

## Some sites also require the user to confirm they are adult for
## adult content. Uncomment by removing '#' in front of is_adult.
is_adult:true

## Don't like the numbers at the start of chapter titles on some
## sites? You can use strip_chapter_numbers to strip them off. Just
## want to make them all look the same? Strip them off, then add them
## back on with add_chapter_numbers. Don't like the way it strips
## numbers or adds them back? See chapter_title_strip_pattern and
## chapter_title_add_pattern in defaults.ini.
#strip_chapter_numbers:true
#add_chapter_numbers:true


[epub]
## Include images from img tags in the body and summary of stories.
## Images will be converted to jpg for size if possible. Images work
## in epub format only. To get mobi or other format with images,
## download as epub and use Calibre to convert.
## true by default, uncomment and set false to not include images.
include_images:true

## If set false, the summary will have all html stripped for safety.
## Both this and include_images must be true to get images in the
## summary.
## true by default, uncomment and set false to not keep summary html.
keep_summary_html:true

## If set true, and there isn't a specific cover image, the first
## image found in the story will be made the cover image. If
## keep_summary_html is true, images in the summary will be before any
## in chapters.
## true by default, uncomment and set false to turn off
make_firstimage_cover:true


## Most common, I expect will be using this to save username/passwords
## for different sites. Here are a few examples. See defaults.ini
## for the full list.

[archiveofourown.org]
username:
password:

## When set true, this setting causes the AO3 code in FFF to replace
## archiveofourown.org with archiveofourown.gay before every network
## request. This does *not* change the story or chapter URLs saved by
## FFF, they will still be archiveofourown.org.

use_archiveofourown_gay:true


## This section will override anything in the system defaults or other
## sections here.
[overrides]
## default varies by site. Set true here to force all sites to
## collect series.
#collect_series: true

[JimmXinu]

Originally Posted by sparrowbby

HTTP Error in FFF '403 Client Error: Forbidden for url: https://archiveofourown.gay/users/login'(403)

Removed the work URL because I don't want to publicly blast what I'm trying to download at random, haha.

The error suggests you have user or pass wrong.

Without a debug log to look at (Calibre Preferences dropdown to restart in debug), or a story URL to try, all I can do is give the link to the ProblematicSites page.

[slimcat]

I'm having an issue with Anthology covers getting removed from the epub on update. Not sure when exactly it started, but it didn't use to happen. I had this issue previously with EPUB3 individual works and you made some updates to the plugin that fixed it. But now I'm seeing a similar issue with EPUB2 Anthologies and I can't figure out why.

I've tested a bunch of different configurations and simplified my personal.ini, but injected anthology covers always get removed. It doesn't make a difference whether I select Update EPUB if New Chapters, Update EPUB Always, or Overwrite Always.

It also doesn't matter whether I set default/site-specific covers in the Generate Cover Plugin settings or set them in my personal.ini and allow generate_cover_settings from personal.ini to override. I don't use Calibre Generate Cover, but I tested it too and epub covers still get removed.

I've got use_old_cover:true in my personal.ini and my generate cover settings are set to:

Generate Calibre Cover: Yes
X Generate Covers Only for New Books
X Inject/update the generated cover inside EPUB

It's happening even with the barebones personal.ini below (and also a version without "generate_cover_settings"). Cover gets injected on initial download but wiped on update:

[defaults]

is_adult:true

collect_series: true

generate_cover_settings:
 ${title} => (?s). => Penguin 3-line Title

use_old_cover:true

[epub]

include_images:true
make_firstimage_cover:false

[archiveofourown.org]

username:XXXXXXXX
password:XXXXXXXX
use_archiveofourown_gay:true

[overrides]

This only happens with Anthologies. Individual stories (both epub 2 and 3) behave as expected.

Any idea what could be going on? You've said before that FFF detects covers using the <reference type="cover"> tag in EPUB2, and it's there before the update. Am I missing something else?

[sparrowbby]

Originally Posted by JimmXinu

The error suggests you have user or pass wrong.

Without a debug log to look at (Calibre Preferences dropdown to restart in debug), or a story URL to try, all I can do is give the link to the ProblematicSites page.

Here's that log:

[JimmXinu]

Originally Posted by slimcat

I've tested a bunch of different configurations and simplified my personal.ini, but injected anthology covers always get removed. It doesn't make a difference whether I select Update EPUB if New Chapters, Update EPUB Always, or Overwrite Always.

The injected part needs to be emphasized here. We're talking about the cover image injected into the epub.


FFF cover support is over complicated. I really should have said 'no' more often in the early days to different cover options. There's a whole wiki page where I try to explain it.

FFF Anthology cover support is even more special and has it's own section in the wiki page.


FFF anthology works by downloading individual stories and using EpubMerge to create the anthology epub. Anthology updates work by using EpubMerge UnMerge, updating the individual stories, then merge together again.

As it exists today, EpubMerge's UnMerge does not give the cover image and FFF doesn't save it. I will think about how feasible it to do so.


In the meantime:

1. Calibre's cover in metadata is unchanged. If you use Calibre to send the epub to a device or save it to disk, Calibre will put the metadata cover inside it by default. For most people the problem you've identified is effectively invisible.
   
2. You can still Polish book to inject the cover, albeit manually.
   
3. If you uncheck "Generate Covers Only for New Books", FFF will generate and inject the cover every time.

[JimmXinu]

Originally Posted by sparrowbby

Here's that log:

I'm able to download that story fine. And I get a different "Failed to Login" error if my user/pass are incorrect.

Unfortunately, I don't have a solution. I suspect you're being blocked, but I couldn't tell you why. Typically, we assume it's for 'too much traffic'.

You can look at the ProblematicSites page about Browser Cache and Proxy features. Some users also have had luck with using a different network--forcing an IP refresh, using a VPN etc.

Worst case, AO3 stories each have a Download button and you can download their version.