[GUI Plugin] FanFicFare

[AbyssalAriel]

Originally Posted by JimmXinu

Using add_to_custom_columns_settings under [archiveofourown.org] for that column will only set it for AO3 stories.

I know I need to put it under [archiveofourown.org], but I'm not sure about the syntax, because it's a Yes/No column.

Like, this works fine:

custom_columns_settings:
 ao3categories=>#ao3_category

But wouldn't using this:

custom_columns_settings:
 status=>#completecol

Just try and fail to send the literal text "Completed" or "In-Progress" to completecol? The custom column menu (that applies to all works) seems to ask me to "check" a specific value and decide yes/no based on that, so I wanted to know how to do that inside the ini file.

Originally Posted by JimmXinu

FFF has no way of knowing when you have edited a Calibre custom column vs FFF previously setting it. If you don't want FFF to ever overwrite your changes, the only way with custom_columns_settings is to use the 'n' mode option so it is only used with new books. See the comments in defaults.ini for custom_columns_settings.

If I only set it the column for AO3 stories though, it won't try to overwrite stories from other sources yeses or nos with "undefined" or some other default, right?

Originally Posted by JimmXinu

The only solution that comes to mind is to:

1. Use add_to_extra_valid_entries and include_in_* to create copies of datePublished, dateUpdated and chapterslashtotal so you aren't affecting default processing that use them, plus your new entry, let's call it mergedentry.
2. Use replace_metadata to add the label to each of those copied entries.
3. Use include_in_mergedentry to put them in one list and keep_in_order_mergedentry:true to suppress list sorting.
4. Use join_string_mergedentry:\s|\s to change the separator.

Should be possible using appropriate regexps in replace_metadata on mergedentry_LIST. There was a discussion earlier this month on something similar.

Thank you, I'll give these a try.

Question, does all this stuff get saved along with the other normal story metadata?

Originally Posted by JimmXinu

You are correct, FFF doesn't currently do that. It could (probably) be done, but at the expense of not being able to order the entries in the list.

Given the absolute mess that AO3 tagging is, I'm not very inclined to pursue this. Especially since it would only save one click--click the story link, then you have all the tags links right there.

The positive answer about the underlining means this doesn't matter much to me anymore (you're right, an extra click makes little difference, and the main goal with the links was to get the underlining TBH), but I'm curious about these two things now:

How would this stop ordering entries in the list?

And how does AO3 tagging being a mess affect it? I wasn't talking about using the "canonical" tag names for the links (that would be quite a mess), just pulling them right out of the page. Or even just using the solution you gave me for underlining to wrap the link around it.

Originally Posted by JimmXinu

No. It's been requested and considered more than once and it just doesn't fit into the plugin.

For repetitive testing, I use FFF's CLI with the --save-cache option to cache the pages once and use them over and over. However, officially, that option is undocumented & unsupported.

Aw, that's a shame. What about it doesn't fit in? Phrasing it that way sounds like it's not a technical barrier, and for design philosophy it doesn't seem any conceptually different from updating calibre's metadata using the same saved data column.

[JimmXinu]

Originally Posted by AbyssalAriel

I know I need to put it under [archiveofourown.org], but I'm not sure about the syntax, because it's a Yes/No column.

Yes/No columns will be set True if the metadata entry is in ('t','true','1','yes','y'), False if in ('f','false','0','no','n') (case insensitive) and set 'None' for tri-state columns when any other value.

So you need to replace Competed and In-Progress to t/f or y/n etc if you want to use a yes/no column. I'd recommend making another extra entry, copy status, etc again before changing it.

Originally Posted by AbyssalAriel

If I only set it the column for AO3 stories though, it won't try to overwrite stories from other sources yeses or nos with "undefined" or some other default, right?

Shouldn't. If it does, I'd like to hear about it.

Originally Posted by AbyssalAriel

Question, does all this stuff get saved along with the other normal story metadata?

No. Only the original collected metadata is saved in the Saved Metadata column, everything is recreated from the original data and the configuration.

Originally Posted by AbyssalAriel

How would this stop ordering entries in the list?

The make_linkhtml_entries feature is a bit of a kludge. It relies on the newentry and newentryUrl lists having corresponding values. Therefore, newentry has to be kept in original order or else the newentryUrl entries won't line up correctly.

Originally Posted by AbyssalAriel

And how does AO3 tagging being a mess affect it?

Mostly because it makes me leery of anything and everything to do with it.

AO3 has 4 different sets of tags which FFF users expect to be separate and stories routinely have dozens or even hundreds of tags. And most users want at least chars & ships sorted.

Originally Posted by AbyssalAriel

Phrasing it that way sounds like it's not a technical barrier, and for design philosophy it doesn't seem any conceptually different from updating calibre's metadata using the same saved data column.

Here are some of the posts from the discussion:
https://www.mobileread.com/forums/sh...01#post3842401
https://www.mobileread.com/forums/sh...postcount=3407
https://www.mobileread.com/forums/sh...postcount=3425

The final conclusion was:

Originally Posted by JimmXinu

I have spent waaay more time (including another hour today) than this corner case can justify trying to fit this into the existing system in way that both works 'as I expect to' and without breaking anything else. I give up.

It's not like there isn't already a way to do it, it's just not as efficient. Use 'Update EPUB always', or don't; it's your choice.

[AbyssalAriel]

Originally Posted by JimmXinu

Yes/No columns will be set True if the metadata entry is in ('t','true','1','yes','y'), False if in ('f','false','0','no','n') (case insensitive) and set 'None' for tri-state columns when any other value.

So you need to replace Competed and In-Progress to t/f or y/n etc if you want to use a yes/no column. I'd recommend making another extra entry, copy status, etc again before changing it.

Okay, sounds easy enough.

Originally Posted by JimmXinu

The make_linkhtml_entries feature is a bit of a kludge. It relies on the newentry and newentryUrl lists having corresponding values. Therefore, newentry has to be kept in original order or else the newentryUrl entries won't line up correctly.

Ah, okay.

Originally Posted by JimmXinu

Here are some of the posts from the discussion:
https://www.mobileread.com/forums/sh...01#post3842401
https://www.mobileread.com/forums/sh...postcount=3407
https://www.mobileread.com/forums/sh...postcount=3425

I see.

I can't help but wonder if I could implement this myself. Call whatever part writes the titlepage normally (and consults the ini files and all) but feed it metadata from the saved column, and then replace the titlepage already in the epub. I wouldn't try getting manually filled columns to work, only the ones that can be done automatically.

If you don't mind saying, what parts of the code would be relevant/useful for this (writeTitlePage and CALIBREONLYSAVECOL look like good leads), and what are the most likely roadblocks?

[JimmXinu]

Originally Posted by AbyssalAriel

I can't help but wonder if I could implement this myself. Call whatever part writes the titlepage normally (and consults the ini files and all) but feed it metadata from the saved column, and then replace the titlepage already in the epub. I wouldn't try getting manually filled columns to work, only the ones that can be done automatically.

If you don't mind saying, what parts of the code would be relevant/useful for this (writeTitlePage and CALIBREONLYSAVECOL look like good leads), and what are the most likely roadblocks?

Generating and injecting the title page are the easy parts. The hard part is fitting it into the existing FFF plugin logically, intuitively, and cleanly.

If you are of a mind to write code, it might be worthwhile to consider either making a general purpose plugin for creating and injecting a metadata jacket (aka title page) from the existing Calibre data, or improving the existing metadata jacket feature in Calibre.

[AbyssalAriel]

I got around to trying the conditional regex stuff with the help of regex101.com, but it's giving me weird results.

Spoiler:

[defaults]
add_to_extra_valid_entries:,datePublishedCopy,dateUpdatedCopy,numChaptersCopy,chapterslashtotalCopy,numWordsCopy,mergedEntry

include_in_datePublishedCopy:datePublished
include_in_dateUpdatedCopy:dateUpdated
include_in_numChaptersCopy:numChapters
include_in_numWordsCopy:numWords
#AO3
include_in_chapterslashtotalCopy:chapterslashtotal 

## Add labels to the copies
add_to_replace_metadata:
 datePublishedCopy=>(.*)=>Published:\s\1
 dateUpdatedCopy=>(.*)=>Completed:\s\1
 numChaptersCopy,chapterslashtotalCopy=>(.*)=>Chapters:\s\1
 numWordsCopy=>(.*)=>Words:\s\1

mergedEntry_label:Stats

## Suppress list sorting
keep_in_order_mergedEntry:true
## Merge together
join_string_mergedEntry:\s|\s

[archiveofourown.org]

add_to_extra_valid_entries:,romships,platships
romships_label:Romantic Ships
platships_label:Platonic Ships

include_in_romships:ships
include_in_platships:ships

# only include / ships in romships and & ships in platships.
# & because HTML
add_to_include_metadata_post:
 romships=~/
 platships=~&

keep_in_order_freeformtags:true
keep_in_order_ao3categories:true
keep_in_order_fandoms:true
keep_in_order_genre:true
keep_in_order_category:true
keep_in_order_characters:true
keep_in_order_ships:true
keep_in_order_romships:true
keep_in_order_platships:true

exclude_metadata_pre:
 freeformtags,genre==Other Additional Tags to Be Added

# Example
# Published: 2022-08-30 | Completed: 2022-08-31 | Chapters: 2/2 | Words: 2,000
include_in_mergedEntry:datePublishedCopy,dateUpdatedCopy,chapterslashtotalCopy,numWordsCopy
add_to_replace_metadata:
 # Delete "Completed" if same date
 mergedEntry_LIST=>(\d{4}-\d{1,2}-\d{1,2})\s\|\sCompleted:\s\1=>\1\s\|
 # Delete chapter count if 1/1
 mergedEntry_LIST=>Chapters:\s1/1\s\|\s=>
 # If complete (same chapter number as total), change chapterslashtotal to just chapter count
 #                 1            2     3   4
 mergedEntry_LIST=>(Chapters:\s)(\d*)/(\2)(\s\|\s)=>\1\2\4
 # If it still has a "Completed", it must have a separate publish date.
 # If it still has a slash, that means it's in-progress.
 # So show "Updated" instead of "Complete".
 mergedEntry_LIST=>Completed:(.*)/=>Updated:\s\1/

titlepage_entries:rating,warnings,ao3categories,fandoms,
 romships,platships,characters,genre,
 bookmarked,bookmarktags,bookmarksummary,
 seriesHTML,series01HTML,series02HTML,series03HTML,collections,
 mergedEntry,description

This is giving me outputs like this on the titlepage:

Published: 2021-05-23Published: | Updated: 2021-05-23Completed: | Chapters: 1/1Chapters: | Words: 2,781Words:

It seems to be adding in extra copies of the labels on the end, and I can't see where I messed up to cause that. It's also a total mystery to me how it's managing to kind of finish the "Updated" part despite all the other junk in the way.

Originally Posted by JimmXinu

Generating and injecting the title page are the easy parts.

Well, maybe it would be easy if I knew my way around the codebase/where things happened, or had looked at python code more recently than 5 years ago

Originally Posted by JimmXinu

The hard part is fitting it into the existing FFF plugin logically, intuitively, and cleanly.

I don't see how fitting it in would be hard, unless by "intuitively" you meant stuff like this sort of user expectation: https://www.mobileread.com/forums/sh...postcount=3407

In any case though, I doubt my attempt will be especially neat or clean.

[dailyalice]

Due to circumstances involving a dead laptop, I'd not used Calibre & FanFicFare in a few years. A few months back, I got back into using it but was doing so on a Chromebook (& couldn't get my personal.ini file to save after editing it) (yes I did check file permissions in various places). (It was an interesting exercise doing things without being able to edit personal.ini, just going with default settings.)

But now I'm able to fully use the latest Calibre & FanFicFare (on a Mac Mini) and have set to work setting up a whole new archive. I've forgotten many things! So as I go, I've been documenting what I do in great detail so I can refer to it myself (because my memory is poor) and possibly share it with others.

As it stands, my guide to FanFicFare and Calibre is at 150+ pages in Google docs. It's getting out of hand and there's so much stuff I haven't covered yet. To be fair, I'm wordy and I'm including screenshots of basically everything.

I have a question that's probably incredibly simple to answer and basic stuff, I'm feeling a bit bad about possibly getting this wrong. And I'm too tired (after hours of working on the guide this morning) to search too much for answers.

I wrote a detailed section of the guide to cover setting up custom columns in personal.ini, only to realize recently that in my instructions I always have folks use (and have screenshots of me using) "add_to_custom_column_settings" when I should probably be using "custom_column_settings" instead.

So far my guide is heavily aimed at folks using AO3 (since that's the site I use the most) and most of my instructions so far for custom columns are specifically for AO3 custom columns. So maybe using "add_to_custom_column_settings" makes sense for the archive specific custom column settings? It does seem to work, but I haven't done extensive testing. And I want to give folks advice on the best way to do things. (Is there any reason it would be better to use "custom_column_settings" than "add_to_custom_column_settings" for AO3 specific columns?)

I'm assuming it makes more sense to use "custom_column_settings" in [defaults] (and perhaps the only way one should handle things in defaults). Fortunately I've only covered one column for defaults so far so I can easily redo that to say "custom_column_settings" instead of "add_to_custom_column_settings."

Thanks in advance.

[JimmXinu]

Originally Posted by dailyalice

...

I hope you'll be willing to share your docs some day. I readily admit I'm not a good documentarian.

When to use add_to_X and when not to: As a general rule, use add_to_X when you want to add settings to more general sections and don't when you want to replace them. So when settings up custom columns for AO3 specific metadata, add_to_ makes perfect sense.

I've attempted to document it's details here:
https://github.com/JimmXinu/FanFicFa...add_to_keyword

[JimmXinu]

Originally Posted by AbyssalAriel

I got around to trying the conditional regex stuff with the help of regex101.com, but it's giving me weird results.

I also need the story URL(s) so I'm looking at the same data.

[AbyssalAriel]

Originally Posted by JimmXinu

I also need the story URL(s) so I'm looking at the same data.

It happens with every URL I've tried, but the one I used for the output I posted here was: https://archiveofourown.org/works/31387700

[dailyalice]

Originally Posted by JimmXinu

I hope you'll be willing to share your docs some day. I readily admit I'm not a good documentarian.

I think you've done a good job within the ini files itself and elsewhere or good enough to get people started. When going through my old personal.ini, I wish I'd thought to add my own comments to explain what I was doing.

I'm already sharing the document with a small audience of folks, some already familiar with FanFicFare & Calibre, some not at all. Happy to share it with more as I go, though I want to get it cleaned up a bit more first. And figure out the best place to publish it.

Anyone know of any other guides out there? Or useful blog posts? I know there have been some blog posts over the years that covered some things, but when I went looking recently I didn't find too much beyond Amalthia's posts on Dreamwidth/Livejournal from way back when (which were helpful then and are still helpful).

I thought more people had shared personal.ini details in posts in the past but I couldn't find as much as I did when I last looked. It's possible they were online and have since vanished.

Originally Posted by JimmXinu

When to use add_to_X and when not to: As a general rule, use add_to_X when you want to add settings to more general sections and don't when you want to replace them. So when settings up custom columns for AO3 specific metadata, add_to_ makes perfect sense.

Thank you! I thought it was something like that, but realized this was the rare case where I started with something I copied from an old personal.ini without looking into why I had things that way.

Originally Posted by JimmXinu

I've attempted to document it's details here:
https://github.com/JimmXinu/FanFicFa...add_to_keyword

Ah yes, I did read this at some point not long ago, just didn't really make the connection when I was covering custom columns. Glad I haven't gone in the wrong direction with the AO3 custom columns. I've been testing out the different bookmark options. (Warning people in my guide that just because you can create columns for all these things, doesn't mean you should! But once I started documenting them, it made sense to cover most things.)


On an unrelated topic: at this point it looks to me like there are potentially three different ways to get word counts (for most folks running Calibre with FanFicFare). Some archives (like AO3) offer counts you can download. And the Page Count plug-in can do word counts (and FanFicFare works with it). And now it sounds like Calibre can do word counts as well (perhaps it was always the case and I didn't notice until recently).

"Calculate word count?" with options "Yes, unless found on site" and "No" and "Yes, always" is in the Basic tab of FanFicFare config in addition to having similar options in the Count Pages tab.

(I think I have "Yes, unless found on site" set both in the Basic Tab and the Count Pages tab. Will that be a problem? Should I say "No" in the Basic tab if I want to use Count Pages to do word counts when word count isn't found on an archive site? I'm guessing it doesn't matter all that much, but it does make me curious which method would get priority or if the software might be off counting words twice or something.)

Thanks so much.

[dailyalice]

Originally Posted by dailyalice

it does make me curious which method would get priority or if the software might be off counting words twice or something.)

I don't mean that word counts would be doubled, just that Calibre & Count Pages might be fighting about it. (Nah, not really. But would Calibre count words and then Count Pages count words as well overwriting what Calibre input?) (I am likely overthinking this.)

[JimmXinu]

Originally Posted by dailyalice

(I think I have "Yes, unless found on site" set both in the Basic Tab and the Count Pages tab. Will that be a problem? Should I say "No" in the Basic tab if I want to use Count Pages to do word counts when word count isn't found on an archive site? I'm guessing it doesn't matter all that much, but it does make me curious which method would get priority or if the software might be off counting words twice or something.)

As I recall, FFF integrated with the Count Pages(CP) plugin first for both word and page counts.

When I later learned that CP calls internal Calibre code to do the word count, I borrowed that from CP to incorporate into FFF so it could be done as part of the FFF background task instead of launching a second background job, CP this time, to generate the same data.

There's no point to doing both--I'd only use CP for word count if also use it for page counts or other statistics.

[JimmXinu]

Originally Posted by AbyssalAriel

It happens with every URL I've tried, but the one I used for the output I posted here was: https://archiveofourown.org/works/31387700

The extra titles can be suppressed by matching string start and stop explicitly:

## Add labels to the copies
add_to_replace_metadata:
 datePublishedCopy=>^(.*)$=>Published:\s\1
 dateUpdatedCopy=>^(.*)$=>Completed:\s\1
 numChaptersCopy,chapterslashtotalCopy=>^(.*)$=>Chapters:\s\1
 numWordsCopy=>^(.*)$=>Words:\s\1

I'm honestly not sure why that's needed sometimes. Logically, it shouldn't be and I believe it changed at one point--pretty confident it wasn't needed when replace_metadata was developed and popped later. A difference between regex in py2 vs py3 maybe?

[dailyalice]

Originally Posted by JimmXinu

As I recall, FFF integrated with the Count Pages(CP) plugin first for both word and page counts.

When I later learned that CP calls internal Calibre code to do the word count, I borrowed that from CP to incorporate into FFF so it could be done as part of the FFF background task instead of launching a second background job, CP this time, to generate the same data.

There's no point to doing both--I'd only use CP for word count if also use it for page counts or other statistics.

Oh excellent, this is good to know. Less stuff for people to worry about, not that setting up the Count Pages plug-in is all that taxing, but if people aren't using any of its features, why waste that time?

Thanks!

[AbyssalAriel]

Originally Posted by JimmXinu

The extra titles can be suppressed by matching string start and stop explicitly:

## Add labels to the copies
add_to_replace_metadata:
 datePublishedCopy=>^(.*)$=>Published:\s\1
 dateUpdatedCopy=>^(.*)$=>Completed:\s\1
 numChaptersCopy,chapterslashtotalCopy=>^(.*)$=>Chapters:\s\1
 numWordsCopy=>^(.*)$=>Words:\s\1

I'm honestly not sure why that's needed sometimes. Logically, it shouldn't be and I believe it changed at one point--pretty confident it wasn't needed when replace_metadata was developed and popped later. A difference between regex in py2 vs py3 maybe?

So that's done the trick, but now I'm having another strange problem (or perhaps it just slipped my notice last time because the other issue was more glaring). On another site, it won't include numWordsCopy, or rather, numWordsCopy seems to just stay empty (but only on this site; AO3 includes it just fine).

Published: 2021-11-23 | Completed: 2021-12-12 | Chapters: 4

Having tested with numWordsCopy in title_entries directly, it looks like the problem is that nothing is being put in there, even though I checked and numWords does have a value.

Code:

Spoiler:

[defaults]

is_adult:true

## Don't affect originals
add_to_extra_valid_entries:,datePublishedCopy,dateUpdatedCopy,numChaptersCopy,chapterslashtotalCopy,numWordsCopy,mergedEntry

include_in_datePublishedCopy:datePublished
include_in_dateUpdatedCopy:dateUpdated
include_in_numChaptersCopy:numChapters
include_in_numWordsCopy:numWords
## AO3
include_in_chapterslashtotalCopy:chapterslashtotal 

## Add labels to the copies
add_to_replace_metadata:
 datePublishedCopy=>^(.*)$=>Published:\s\1
 dateUpdatedCopy=>^(.*)$=>Completed:\s\1
 numWordsCopy=>^(.*)$=>Words:\s\1
 numChaptersCopy,chapterslashtotalCopy=>^(.*)$=>Chapters:\s\1

mergedEntry_label:Stats

## Suppress list sorting
keep_in_order_mergedEntry:true
## Merge together
join_string_mergedEntry:\s|\s

#################################################
#################################################

[literotica] # header was edited for this forum, just in case it counts as a "link"

keep_in_order_eroticatags:true
keep_in_order_genre:true
include_in_genre:eroticatags
include_subject_tags:genre

extratags:
extra_titlepage_entries:

# Published: 2022-08-30 | Completed: 2022-08-31 | Chapters: 2 | Words: 2,000
include_in_mergedEntry:datePublishedCopy,dateUpdatedCopy,numChaptersCopy,numWordsCopy
add_to_replace_metadata:
# Delete "Completed" if same date
 mergedEntry_LIST=>(\d{4}-\d{1,2}-\d{1,2})\s\|\sCompleted:\s\1=>\1
# Delete chapter count if oneshot
 mergedEntry_LIST=>Chapters:\s1\s\|\s=>

titlepage_entries:seriesHTML,
 rating,warnings,category,
 ships,characters,genre,
 mergedEntry,description

chapter_categories_use_all:true
clean_chapter_titles:true
description_in_chapter:true

I would have included an example of a URL having this problem (although really it's all of them from this site, both oneshots and series), but I believe I read somewhere that links to sites like literotica aren't allowed on the mobileread forum?