[GUI Plugin] FanFicFare

[tectonictigress]

Hello! First off - thank you for everything you do with managing this plug-in, it is truly great, and I've visited this forum a few times in the past to get tips or find similar questions to my own

I was hoping to get some guidance on what I may be doing wrong with some metadata management I'm trying to do, specifically tag wrangling with AO3

For example, there is a tag labeled "2k words of angst" that I am trying to remove by removing all tags that start with a number

exclude_metadata_post:
 genre=~^[0-9]

This works when I test it with test1.com, however for the actual AO3 page I am trying to bring in, it doesn't work - I have this issue with several other tags and fics I am trying to filter, so I don't think it's the regex that's the issue, but perhaps something with the ordering of my personal.ini? I tried re-arranging it a few times but nothing worked, this is the simplified order it is currently in with test1.com commented out

[defaults]
is_adult:true
[epub]

[archiveofourown.org]

#[test1.com]
#extragenres: 2k words of angst
#extracharacters: Peter, Bruce, Barry, Tom

[overrides]

exclude metadata_post:
 genre=~^[0-9]

May be related - but when using test1.com and re-trying some of this, the tags aren't being updated - I had a much longer list of tags I had listed out after extragenres, but when I cut it down to "2k words of angst" (and uncommented) and re-ran it, it still has the very long list in the tags

I then added extracharacters: to see if it would update, and when I re-ran it, it added the characters, but still has the old list of tags

Is there something wrong in my settings perhaps?

I do not have "Keep Existing Tags when Updating Metadata?" checked on

---------------------------------------------------------------------------------

Separate question

I have a custom column named #time for timelines

I'd like to extract only the tags that start with "Post" from the tag list and move it to the custom column, and apply separate filtering to the genres vs time columns, but I'm have trouble following how to set up personal.ini to do this and how to reference the custom vs genres column

I tried following what you went through with this thread by adding what's noted below, but it had no affect on the custom column

add_to_extra_valid_entries:,time
include_in_time:genre

exclude_metadata_post:
 time=~^Post

---------------------------------------------------------------------------------

Separate question (sorry, didn't realize I had a few lined up I was wanting to ask about)

This is in relation to wrangling ships

Some fics are tagged as "Harry & Ron" and others as "Ron & Harry" - is there a way to write this into personal.ini to align them into the same tag?

Currently I'm using:

replace_metadata:
 ships=>Ron & Harry=>Harry & Ron

but having to do it for each pairing is a bit tedious - I'm targeting to sort them alphabetically - I'm familiar with coding in R and Python and believe I could write something within there to do this would be relatively straightforward by separating the string by "&" or "/", then alphabetizing, then re-adding the "&" or "/" - but I'd rather this be within personal.ini directly

Last question for now with regards to ships:

This is something that's a personal irk, but I like all the names to be capitalized in the ship listing - so far it's just one ship that is doing this (the individual names are actually supposed to be capitalized, and are in other ships listings, just not this particular one for some reason)

I tried manually replacing it in personal.ini with

replace_metadata:
 ships=>Olivia dunham & Peter Bishop=>Olivia Dunham & Peter Bishop

but it just doesn't take and the tag stays as "Olivia dunham & Peter Bishop" - is there something related to regex I'm not understanding with regards to case sensitivity?

Sorry for all the questions at once, I really appreciate any guidance you can provide!

[JimmXinu]

New Test Version Posted

2020-10-25
- Fix for Xenforo2 change (SB/SV)

[JimmXinu]

Originally Posted by tectonictigress

Hello! First off - thank you for everything you do with managing this plug-in, it is truly great, and I've visited this forum a few times in the past to get tips or find similar questions to my own

...

Sorry for all the questions at once, I really appreciate any guidance you can provide!

Umm, okay. Wow. That's a lot.

First, without example story URLs that exhibit the problems, all I can do is speculate. Whereas, if I have actual story URLs, I can try it and see the issue.

That's a general rule around here--doesn't matter what you want help with--I want a story URL that exhibits the issue.

Second, you apparently have a quite complex metadata setup. You may think you've isolated the parts that cause your issues--and you may well be right.

However, if you're going to come to with five different, complex issues at once, I'd like to have your complete personal.ini to test with. You can use the 'View "Safe" personal.ini' button to automatically strip your user/pass settings.

Alternatively, if you are strongly opposed to sharing your settings, I will try a stripped down version, if you confirm that it still exhibits the issues described.

Third, AO3 tag normalizing/wrangling comes up over and over again. I'm not going to go over all of it again, but AO3's decision to allow authors to input basically anything--and any number of anything--as tags makes normalizing problematic at best.

Personally, I consider it futile. But I will help with specific problems.

[tectonictigress]

Originally Posted by JimmXinu

Umm, okay. Wow. That's a lot.

First, without example story URLs that exhibit the problems, all I can do is speculate. Whereas, if I have actual story URLs, I can try it and see the issue.

That's a general rule around here--doesn't matter what you want help with--I want a story URL that exhibits the issue.

Second, you apparently have a quite complex metadata setup. You may think you've isolated the parts that cause your issues--and you may well be right.

However, if you're going to come to with five different, complex issues at once, I'd like to have your complete personal.ini to test with. You can use the 'View "Safe" personal.ini' button to automatically strip your user/pass settings.

Alternatively, if you are strongly opposed to sharing your settings, I will try a stripped down version, if you confirm that it still exhibits the issues described.

Third, AO3 tag normalizing/wrangling comes up over and over again. I'm not going to go over all of it again, but AO3's decision to allow authors to input basically anything--and any number of anything--as tags makes normalizing problematic at best.

Personally, I consider it futile. But I will help with specific problems.

I agree it pretty much is futile, but it's mainly the ships and a handful of certain tags I'm focused on cleaning up a bit - I'm sure it does come up quite often, so I tried my best to go through your past replies and your wiki on github, and did some regex learning so that I wouldn't be wasting your time with regex questions, but alas this is where I got to, so I do appreciate any guidance you can provide

I actually have been using a stripped down personal.ini, as that was the first I thing I used to start testing with when trying to track down the issues

Working through the first question a bit more, I believe I did track down the issue - I was using "Update Existing Fanfiction Books" - this works on fics that still have tags remaining after the updated filtering in personal.ini is applied, but the fics I was having problems with should have had no tags after the filtering is applied - I then deleted the fic, and re-brought it in and it was filtered correctly with no tags, this was also the case for using test1.com

Deleting and re-adding also worked for the capitalization issue for ships, even though I wasn't trying to apply any filtering or be left with zero tags in that case

Not sure if that's what is supposed to happen or if there was something obvious I was missing regarding that issue, so I apologize if that was the case - if you'd still like an example I can provide one

For the custom column question - here is an example fic: https://archiveofourown.org/works/19784374 - and below is the personal.ini, similar to before - mainly just looking for any guidance on how to properly be referencing custom columns in personal.ini

[defaults]

is_adult:true

[epub]

[archiveofourown.org]

[overrides]

include_subject_tags: genre 

add_to_extra_valid_entries:,time
include_in_time:genre

exclude_metadata_post:
 time=~^Post

For aligning ship names - this one (https://archiveofourown.org/works/21923320) has Michelle Jones/Peter Parker - and this one (https://archiveofourown.org/works/18638395) has Peter Parker/Michelle Jones - was just curious if it was possible to parse and alphabetize the tag to be more efficient, but otherwise I can just do it manually as below

[defaults]

is_adult:true

[epub]

[archiveofourown.org]

[overrides]

include_subject_tags: genre 

replace_metadata:
  ships=>Peter Parker/Michelle Jones=>Michelle Jones/Peter Parker

I tried to edit a final question regarding version updating right after I previously posted, but I don't think it went through - I cannot update beyond Calibre 3.48 due to OS limitations, and I am currently using FFF 3.11.0 - are there any concerns I should have regarding updating FFF to newer versions, or will it remain backwards compatible? - If this is something you mention in release notes posted somewhere each time, could you please point me to a link where? I've seen the release notes mentioned here in posts, but wasn't sure if there was somewhere like a github page I can reference for the most recent version

[JimmXinu]

Originally Posted by tectonictigress

Working through the first question a bit more, I believe I did track down the issue - I was using "Update Existing Fanfiction Books" - this works on fics that still have tags remaining after the updated filtering in personal.ini is applied, but the fics I was having problems with should have had no tags after the filtering is applied - I then deleted the fic, and re-brought it in and it was filtered correctly with no tags, this was also the case for using test1.com

I would also comment that rather than deleting the book, you could probably clear the tags field. Since you didn't give any example story URLs, I assume you're happy with your solution.

Originally Posted by tectonictigress

Deleting and re-adding also worked for the capitalization issue for ships, even though I wasn't trying to apply any filtering or be left with zero tags in that case

FFF does a filter to make sure it doesn't try to add the same tag with different letter case, but only when 'Keep existing tags' is checked.

Calibre, on the other hand, keeps a database table of tag values for the whole library; and it only wants you to have one 'case' version. If you manually change it, Calibre appears to change the case for that tag on all books. But if you do it through FFF, calibre will map it to the same case as it's existing tag.

Originally Posted by tectonictigress

For the custom column question - here is an example fic: https://archiveofourown.org/works/19784374 - and below is the personal.ini, similar to before - mainly just looking for any guidance on how to properly be referencing custom columns in personal.ini

You can only set extra_valid_entries to custom columns using the personal.ini custom_columns_settings setting.

From (plugin-)defaults.ini:

Spoiler:

## (Plugin Only) - You can also populate calibre custom columns with
## the site specific metadata using custom_columns_settings (but only
## if 'Allow custom_columns_settings from personal.ini' is checked in
## the plugin GUI config.)  There are three parts, the entry name,
## then the label of the calibre custom column, then (optionally) a
## 'mode'.  'r' to Replace any existing values, 'a' to Add to existing
## value (use with tag-like columns), and 'n' for setting on New books
## only.  (Default is 'r'.)

## Literal strings can be set into custom columns using double quotes.
## Each metadata=>column mapping must be on a separate line and each
## needs to have one space at the start of each line.

## 'r_anthaver' and 'n_anthaver' can be used to indicate the same as
## 'r' and 'n' for normal downloads, but to average the metadata for
## the differents story in an anthology before setting in integer and
## float type custom columns.  This can be useful for a averrating
## column, for example.  Default is to sum the values of all stories,
## and numChapters and numWords are always summed.

#custom_columns_settings:
# cliches=>#acolumn
# themes=>#bcolumn,a
# timeline=>#ccolumn,n
# "FanFiction"=>#collection
# averrating=>#averrating,r_anthaver

Originally Posted by tectonictigress

For aligning ship names - this one (https://archiveofourown.org/works/21923320) has Michelle Jones/Peter Parker - and this one (https://archiveofourown.org/works/18638395) has Peter Parker/Michelle Jones - was just curious if it was possible to parse and alphabetize the tag to be more efficient, but otherwise I can just do it manually as below

Set sort_ships:true

From (plugin-)defaults.ini:

Spoiler:

## Reorder ships so b/a and c/b/a become a/b and a/b/c. '/' is no
## longer hard coded and can be changed and added to with
## sort_ships_splits.
sort_ships:false

## Each line indicates first a regex that should be used to split each
## ships entry and then, after => the string to use to merge the parts
## back together.  \s == blank space.
## Each part will have replace_metadata with key ships_CHARS applied.
sort_ships_splits:
 [ ]*/[ ]*=>/
 [ ]*&[ ]*=>\s&\s

Originally Posted by tectonictigress

I cannot update beyond Calibre 3.48 due to OS limitations, and I am currently using FFF 3.11.0 - are there any concerns I should have regarding updating FFF to newer versions, or will it remain backwards compatible? - If this is something you mention in release notes posted somewhere each time, could you please point me to a link where? I've seen the release notes mentioned here in posts, but wasn't sure if there was somewhere like a github page I can reference for the most recent version

I only support the latest version of FanFicFare. There is usually a more recent test version available, too; see link in my sig.

I will make some effort to support the last minor of each major calibre version back to v2 (so v2.85.1, v3.48 and v4.23).

However, I don't make it a habit of testing against older versions (or OS's other than Win10) unless I have reason to do so, like a specific complaint.

Unless a user tells me otherwise, I assume everyone is using the latest FFF on the latest Calibre.

Yes, there is a github repo with all the past releases:
From the first post of the thread:

Originally Posted by JimmXinu

This plugin brings the functionality of FanFicFare to Calibre. The source is available on the GitHub project home page. There's additional documentation and FAQs on the project wiki, including a page specifically about the plugin version.

[ownedbycats]

Originally Posted by JimmXinu

I would also comment that rather than deleting the book, you could probably clear the tags field. Since you didn't give any example story URLs, I assume you're happy with your solution.

I can confirm that clearing the tags (except for the url identifier) always seems to work, so long as none of the column updates are set to "new only."

[tectonictigress]

Originally Posted by JimmXinu

I would also comment that rather than deleting the book, you could probably clear the tags field. Since you didn't give any example story URLs, I assume you're happy with your solution.

FFF does a filter to make sure it doesn't try to add the same tag with different letter case, but only when 'Keep existing tags' is checked.

Calibre, on the other hand, keeps a database table of tag values for the whole library; and it only wants you to have one 'case' version. If you manually change it, Calibre appears to change the case for that tag on all books. But if you do it through FFF, calibre will map it to the same case as it's existing tag.

Yes, I am happy with figuring out the solution for this issue - I still don't entirely understand why needing to clear out the existing tags is necessary and how updating vs initial import handles tags differently, but I can read through the scripts and documentation in more detail and perhaps find more info there if I really want to know - the explanation regarding how FFF vs Calibre handles tag helps with understanding the 'case' issue, thank you

You can only set extra_valid_entries to custom columns using the personal.ini custom_columns_settings setting.

From (plugin-)defaults.ini:

Spoiler:

## (Plugin Only) - You can also populate calibre custom columns with
## the site specific metadata using custom_columns_settings (but only
## if 'Allow custom_columns_settings from personal.ini' is checked in
## the plugin GUI config.)  There are three parts, the entry name,
## then the label of the calibre custom column, then (optionally) a
## 'mode'.  'r' to Replace any existing values, 'a' to Add to existing
## value (use with tag-like columns), and 'n' for setting on New books
## only.  (Default is 'r'.)

## Literal strings can be set into custom columns using double quotes.
## Each metadata=>column mapping must be on a separate line and each
## needs to have one space at the start of each line.

## 'r_anthaver' and 'n_anthaver' can be used to indicate the same as
## 'r' and 'n' for normal downloads, but to average the metadata for
## the differents story in an anthology before setting in integer and
## float type custom columns.  This can be useful for a averrating
## column, for example.  Default is to sum the values of all stories,
## and numChapters and numWords are always summed.

#custom_columns_settings:
# cliches=>#acolumn
# themes=>#bcolumn,a
# timeline=>#ccolumn,n
# "FanFiction"=>#collection
# averrating=>#averrating,r_anthaver

Ok, so custom_columns_settings essentially initiates the column 'variable' that I can then apply filtering to - got it to work now, thank you

Set sort_ships:true

From (plugin-)defaults.ini:

Spoiler:

## Reorder ships so b/a and c/b/a become a/b and a/b/c. '/' is no
## longer hard coded and can be changed and added to with
## sort_ships_splits.
sort_ships:false

## Each line indicates first a regex that should be used to split each
## ships entry and then, after => the string to use to merge the parts
## back together.  \s == blank space.
## Each part will have replace_metadata with key ships_CHARS applied.
sort_ships_splits:
 [ ]*/[ ]*=>/
 [ ]*&[ ]*=>\s&\s

Got it, didn't realize that was buried in the default, and it works on & as well, thanks for pointing it out!

I only support the latest version of FanFicFare. There is usually a more recent test version available, too; see link in my sig.

I will make some effort to support the last minor of each major calibre version back to v2 (so v2.85.1, v3.48 and v4.23).

However, I don't make it a habit of testing against older versions (or OS's other than Win10) unless I have reason to do so, like a specific complaint.

Unless a user tells me otherwise, I assume everyone is using the latest FFF on the latest Calibre.

Yes, there is a github repo with all the past releases:
From the first post of the thread:

Yes, I know about the repo with past releases - I was asking about an updated page that mentioned major changes that may affect version compatibility, or any other major announcements regarding updates, it doesn't sound like that exists so I will just reference the repo and revert to a previous version if I come across any issues that may be related to an update

Thank you for creating and providing this tool, I appreciate the quick responses

[ownedbycats]

replace_metadata: 
## Removal of challenge series
 series=>(Whumptober .*)=>

This leaves a blank "Series:" on the title page. Should I have done this some other way?

[JimmXinu]

Originally Posted by tectonictigress

Yes, I am happy with figuring out the solution for this issue - I still don't entirely understand why needing to clear out the existing tags is necessary and how updating vs initial import handles tags differently, but I can read through the scripts and documentation in more detail and perhaps find more info there if I really want to know - the explanation regarding how FFF vs Calibre handles tag helps with understanding the 'case' issue, thank you

Did you say that you'd cleared all values for that metadata entry? FFF doesn't set custom columns to a metadata value if it's empty. I don't remember the exact reasons why it works that way.

Originally Posted by tectonictigress

Ok, so custom_columns_settings essentially initiates the column 'variable' that I can then apply filtering to - got it to work now, thank you

Not exactly.

If you want to use the programming metaphor, you declare a new variables with extra_valid_entries. You can assign to them with include_in_varname and change the values with replace_metadata and the in/exclude settings. custom_columns_settings can then assign those variables to custom calibre columns by calibre 'lookup/search name'. All calibre custom column lookup names begin with '#'.

Originally Posted by tectonictigress

Yes, I know about the repo with past releases - I was asking about an updated page that mentioned major changes that may affect version compatibility, or any other major announcements regarding updates, it doesn't sound like that exists so I will just reference the repo and revert to a previous version if I come across any issues that may be related to an update

There's a version history in the first post of the thread, there are posts in the thread for every new version (and test version), and release notes for every release in the repo.

The last time there was a change that definitely was effected by calibre version was noted:

Version 3.15.0 - 15 Jan 2020
- Remove site nfacommunity.com, parked domain, last worked Dec2018.
- Remove site fannation.shades-of-moonlight.com, parked domain, last worked Dec2018.
- Kludge fix for incorrect HTTP response 500 code from trekfanfiction.net.
- trekfanfiction.net uses utf8 now
- Make sure storyUrl has &->& and then change it back for Calibre. Matches how authorUrl is handled(in all_metadata). Closes #460 for adapter_sugarquillnet .
- adapter_sugarquillnet change chapter text td to div. Issue #460
- Update adapter_alternatehistorycom for XenForo2. Issue #457
- Order custom columns in config page.
- Add -U/--update-epub-always option to CLI.
- Don't do URL quoting on file: URLs. Fix for spaces->+ breaking default cover.
- Catch ',' in front of 'Thread' in forum titles. XenForo sites.
- Set minimum_calibre_version to v2.85.1, last of 2 series.
- Changes for Python3 compatibility, thanks eschwartz!

Originally Posted by ownedbycats

replace_metadata: 
## Removal of challenge series
 series=>(Whumptober .*)=>

This leaves a blank "Series:" on the title page. Should I have done this some other way?

Title page actually uses seriesHTML. Clear that, too, and it should work.

Calibre, it seems, doesn't clear the standard series column if set to empty string('') by FFF.

[ownedbycats]

## Removal of challenge series
 series=>(Whumptober .*)=>
 seriesHTML=>(Whumptober .*)=>

Still left a blank Series: on the title page of this fic. It did clear it from the Calibre metadata upon initial download though.

[JimmXinu]

Well, you've yet again found another weird little corner case. Congratulations!

Title page uses seriesHTML, which FFF constructs as an HTML when it finds that it has a seriesUrl value, or just text w/o link if there's a series, but no seriesUrl:

linkhtml="<a class='%slink' href='%s'>%s</a>"

if self.getMetadataRaw('seriesUrl'):
    self.setMetadata('seriesHTML',linkhtml%('series',
                                            self.getMetadata('seriesUrl', removeallentities, doreplacements),
                                            self.getMetadata('series', removeallentities, doreplacements)))
elif self.getMetadataRaw('series'):
    self.setMetadata('seriesHTML',self.getMetadataRaw('series'))

Because you cleared series, but the code looks for raw seriesUrl, seriesHTML still contains HTML, just not visible HTML.

Trying to clear seriesHTML based on the text value fails, because 'Whumptober' was already removed.

I will think about whether the code should be changed--IE, what else might break if I fix this issue in code.

In the meantime, this works:

 seriesHTML=>.*=>&&seriesUrl=>https://archiveofourown.org/series/1951183
 series=>.*=>&&seriesUrl=>https://archiveofourown.org/series/1951183

[ownedbycats]

Thanks. If it doesn't break anything else I'd appreciate a fix. Until then since challenge fics tend to be oneshots I can just manually fix it with ebook-editor.

[JimmXinu]

New Test Version Posted

2020-10-25
- Change seriesHTML to not be filled without series.
- Fix for Xenforo2 change (SB/SV)

[ownedbycats]

Confirm that the new seriesHTML fix worked. Thank you.

[tectonictigress]

Originally Posted by JimmXinu

Did you say that you'd cleared all values for that metadata entry? FFF doesn't set custom columns to a metadata value if it's empty. I don't remember the exact reasons why it works that way.

No, I hadn't cleared anything from how the metadata was originally brought in, I was just updating personal.ini and the fic, and checking Update Calibre Metadata - this was just for the Tags column read in by default, not for a specific custom column

There's a version history in the first post of the thread, there are posts in the thread for every new version (and test version), and release notes for every release in the repo

Yes, as I said I know about the release notes, the one in the first post is to me not obvious that the minimum_calibre_version comment relates to version compatibility, as someone who is new to this particular context - I was more or less hoping something regarding the version compatibility would be mentioned as part of the README or more overt on first post in the thread, rather than having to search through release notes, similar to how Calibre notes the required version based on the OS version on their download page (https://calibre-ebook.com/download_osx) - thank you for clarifying what I should look for in the future