[GUI Plugin] FanFicFare

[JimmXinu]

Adding labels to all values for a metadata entry (ie, genre) is straightforward, but would affect everywhere it appears.

Doing it for only for one specific column, but not everywhere, is more complex.

I know there are users who do similar things for Calibre's nesting tags. Perhaps one of them can help before we re-invent this particular wheel?

[ThreeLizards]

I tried doing a column from other columns column but the catalog didn’t like using that one, which sucks cuz I had one perfect

So hopefully the others have a suggestion

[culytera]

Originally Posted by ThreeLizards

Hi!

So I make catalogs of the fanfics that I upload to my kindle, make it easy to find books. It lets you choose a tags column. Well, the column I was using, #mytags, didn’t have fandom or pairing in it and I want to add those into a column so they’ll be in the catalog. So I created a #catalog_column and set up in the personal INI to have the appropriate data sent there. However, I would like it if I could have it modify the info, so instead of Marvel, it’ll put Fandom: in front of it, that way all the fandoms are sorted together. Is there a way to set this up in the personal ini?

TLDR
I have:
CatalogColumn shows Marvel, Stony, T

I want:
CatalogColumn: Fandom: Marvel, Pairing: Stony, Rating: T

Thank you for your help!

I do something similar with the tags, though I don't add these to a column so you probably need to use add_to_custom_columns_settings instead:

add_to_extra_valid_entries:,prefix_category,prefix_rating,prefix_ships,prefix_status
include_in_prefix_category:category
include_in_prefix_rating:rating
include_in_prefix_ships:ships
include_in_prefix_status:status
keep_in_order_prefix_category:true
keep_in_order_prefix_ships:true

replace_metadata:
 prefix_ships=>^(.*)=>[p] \1
 prefix_category=>^(.*)=>[c] \1
 prefix_rating=>^(.*)=>[r] \1
 prefix_status=>^(.*)=>[s] \1

include_subject_tags:prefix_ships_LIST,prefix_category_LIST,prefix_rating,prefix_status

[crimson]

Originally Posted by JimmXinu

The easiest way to add newlines to custom output is to make it multi-line in INI:

titlepage_entry:
 <b>${label}:</b> ${value}<br />

I don't know an easy way to get multiple empty newlines off hand.

Look at the newlines in txt for multiple empty newlines. Works the same in HTML.

# make sure to keep a single space on the line after this--it's
# one, empty newline for inclusion below.  Define in same section as entry.
newline:
 
# You can't see it, but there's a space on the line above.

titlepage_entry:
 <b>${label}:</b> ${value}<br />
 %(newline)s

I am unable to use %(newline)s. It says:

InterpolationMissingOptionError: Bad value substitution: option 'titlepage_entry' in section 'epub' contains an interpolation key 'newline' which is not a valid option name. Raw value: '<b>${label}:</b> ${value}<br />\n%(newline)s'

[JimmXinu]

Originally Posted by crimson

I am unable to use %(newline)s. It says:

InterpolationMissingOptionError: Bad value substitution: option 'titlepage_entry' in section 'epub' contains an interpolation key 'newline' which is not a valid option name. Raw value: '<b>${label}:</b> ${value}<br />\n%(newline)s'

That suggests you skipped added this part:

# make sure to keep a single space on the line after this--it's
# one, empty newline for inclusion below.  Define in same section as entry.
newline:
 
# You can't see it, but there's a space on the line above.

[JimmXinu]

Originally Posted by culytera

I do something similar with the tags, though I don't add these to a column so you probably need to use add_to_custom_columns_settings instead:

add_to_extra_valid_entries:,prefix_category,prefix_rating,prefix_ships,prefix_status
include_in_prefix_category:category
include_in_prefix_rating:rating
include_in_prefix_ships:ships
include_in_prefix_status:status
keep_in_order_prefix_category:true
keep_in_order_prefix_ships:true

replace_metadata:
 prefix_ships=>^(.*)=>[p] \1
 prefix_category=>^(.*)=>[c] \1
 prefix_rating=>^(.*)=>[r] \1
 prefix_status=>^(.*)=>[s] \1

include_subject_tags:prefix_ships_LIST,prefix_category_LIST,prefix_rating,prefix_status

This is largely what I was thinking as the 'more complex' version. Create new metadata entries for prefix versions, add prefix, then use.

As culytera said, you'll need custom_columns_settings to put them in a column, but it sounded like you already had that part.

However, custom_columns_settings only allows for one metadata entry per line, so you'll need to either merge the different prefix_X into another new entry first or use multiple custom_columns_settings lines with 'a' flags.

It looks like culytera uses the keep_in_order_X settings and X_LIST to retain author's ordering for those entries. That may or may not work with custom column , depending on your custom column type and how you merge the different entries.

[ownedbycats]

Question: Is there a way to include_metadata on a certain fandom if it's the only one on the story?

For example, a lot of Star Trek: Original Series are also double/triple-tagged Star Trek: Strange New Worlds and/or Alternate Original Series. (Here's a bunch of examples with all three.)

I can just exclude them, but for futureproofing I'd prefer to filter if it's single-category (actual crossovers can be dealt with in story-sections).

[JimmXinu]

I don't understand the question. If story has category tags:

Star Trek: Alternate Original Series (Movies)
Star Trek: The Original Series
Star Trek: Strange New Worlds (TV)

What is it you want that story to have?

[ownedbycats]

The Original Series. I only want AOS or SNW if they're the single category.

[JimmXinu]

Change them all to 'Star Trek' and call it done? AO3 tags:

[JimmXinu]

Experimental Version Attached

2025-05-17
- Single proc bg processing, optionally split by site & accumulate results -- experimental

2025-05-18
- Improve job 'reconsolidate' for failed jobs and setting changing.

Background Job Processing in Past
When first implementing this downloader as a plugin in 2011, I copied the background processing code from Extract ISBN. That code, after launching as a Calibre background job, spawns another level of sub-processes, one per book, running as many processes in parallel at once as Calibre's config allow--generally as many as there are CPUs/cores.

Back in 2021, we finally realized that was allowing stories from the same site to run simultaneously, causing some sites to block users for excessive traffic. I changed FFF to instead run one background process per site, IE, only running one download from each site at a time, but still allowing parallel downloads from different sites.

Something that had been occasionally reported was that download jobs would not end when their parent Calibre job was stopped, or even when Calibre was closed. (At least on Windows.) I confess I didn't really pay them much attention until use of open_pages_in_browser made it more obvious something was going on.

Background Job Processing in This Version
The attached version has three additional optional settings For this first experimental version the settings are on the 'Other' tab of FFF Config with some explanatory verbiage.

I'm not entirely decided on which of these should be hard coded, optional settings, grouped as one setting, or what the defaults values should be. Opinions welcome.

The options are:

1. Use New Single Process Background Jobs
   On by default. Don't spawn additional sub-processes from the Calibre background job; instead all downloads in the job are processed sequentially, more like how CLI does. With only this checked, the only difference from prior versions is some slow down when downloading stories from more than one site together in one job. Uncheck this to use the old multi-process code from before.
2. Split downloads into separate background jobs by site
   Off by default. Split different sites into separate processes that can run in parallel, but they each run their own Calibre Job. You can see the different Jobs for different sites in Calibre's Job list. Because Calibre limits the number of simultaneous Jobs running, only a few will be running. This was true before as well, but it wasn't visible to the user. As each Job finishes, you will be prompted to update your library for each site separately, unless you also set:
3. Reconsolidate split downloads before updating library
   Off by default. Only useful when Split downloads is also checked. When checked, FFF keeps track of which BG Jobs launched together and will wait until they all finish before prompting you to update your library.

Disadvantages of new version

• Downloads from different sites only done in parallel if you also check 'Split downloads...'.
• If split, you will get a separate 'Proceed to update library' question for each site, unless you also check 'Reconsolidate split downloads...'.

Advantages of new version

• Download job actually stops when job is stopped or Calibre quits. No more open_pages_in_browser calls after you've quit Calibre.
• Job Details (aka Job log) updates real time, you can watch downloads in progress.
• Job start is quicker by several seconds.
• 'Split' without 'Reconsolidate' allows library updates sooner for sites that finish sooner.

Related Questions

• Why not skip the 'Proceed with updating your library' question?
  The Count Pages and Extract ISBN plugins, for example, allow the user to optionally skip the 'Proceed' question. I did revisit that while working on this and even tried it for a while.
  
  The problem is that FFF's post-processing can be significant, and it uses a progress bar dialog. When multiple jobs finished, they would start updating simultaneously and stacking nearly identical progbars--and it could happen with manually started parallel updates, not just 'Split' jobs. Visually, it looked like one progbar jumping back and forth unexpectedly.
  
  Rather than discovering what in FFF is and isn't safe for parallel access, I decided to keep the Proceed question. Which also follows recommended Calibre practice.
• Why not use threads in the background process to allow parallel downloads?
  I did try this as well. Basic testing seemed to work, but the log output for the different threads came out all interspersed. As someone who needs to read those logs a lot for troubleshooting, that was a deal breaker.
  
  And again, FFF has been running basically single threaded this whole time. I don't think it would be a matter of if it caused problems, but when.
• How does this affect Anthologies?
  Anthology stories will download/update sequentially in one Job, ignoring the 'Split' option.
  
  For anthologies of stories all from one site (I assume the vast majority of anthologies), it will actually be slightly faster, since the sub-proc launch time is saved.
  
  For rarer anthologies of stories split between different sites, it will be slower, since there won't be any parallel downloads.

UPDATE:
2025-05-18
- Improve job 'reconsolidate' for failed jobs and setting changing.

[ownedbycats]

Sounds good. But I'm curious why users are closing Calibre in the middle of a download? This seems to be when the job is running, so it's not the nonresponsive cancel button on the initial popup.

[JimmXinu]

Well, when I've done it, it's been because of accidentally leaving open_pages_in_browser, check_next_chapter, and more recently, try_shortened_title_urls all on when downloading an ffnet story that doesn't have a hidden next chapter.

[JimmXinu]

Experimental Version Updated

2025-05-18
- Improve job 'reconsolidate' for failed jobs and setting changing.

I've updated the Experimental Version previously posted so job reconsolidation works if a job fails entirely, or if the user turns that option off after launching a split job.

If no problems come up, I will probably promote this version to 'Testing' tomorrow.

I'm still unsure which of the current three options should be kept optional, what their default state should be, or where they should be in FFF's Config. If anyone has opinions, I'd be interested to hear them.

[JimmXinu]

New Test Version Posted

2025-05-19
- Single proc bg processing, optionally split by site & accumulate results (See this post)