[GUI Plugin] FanFicFare

[culytera]

I'm trying to understand why my test entries aren't being used despite including only them as subject tags.

Basically what I'm trying to do is retrieving the first item in a list. I've managed to do this (maybe??) as shown on the title page, but for whatever reason it's taking other entries as subject tags. I didn't see any conflicting settings on overrides/defaults. Here's the relevant code:

Spoiler:

[test1.com]
# URL: http://test1.com?sid=12345

extra_valid_entries:prefix_category,prefix_ships,first_category,first_ship

include_in_prefix_ships:ships
include_in_prefix_category:category

include_in_first_ship:prefix_ships
include_in_first_category:prefix_category

keep_in_order_ships:true
keep_in_order_characters:true
keep_in_order_category:true
keep_in_order_prefix_ships:true
keep_in_order_prefix_category:true
keep_in_order_first_ship:true
keep_in_order_first_category:true

prefix_ships_label:Prefix Ships
prefix_category_label:Prefix Category
first_ship_label:First Ship
first_category_label:First Category

exclude_metadata_pre:
 prefix_ships=~(.*) & (.*)

replace_metadata:
 prefix_category=>^(.*)=>[c] \1
 prefix_ships=>^(.*)=>[p] \1
 first_category_LIST=>^(\[c\].*?)(\,.*)=>[first] \1
 first_ship_LIST=>^(\[p\].*?\/.*?)(\,.*)=>[first] \1

titlepage_entries:category,characters,ships,prefix_ships,prefix_category,first_ship,first_category
include_subject_tags:first_ship,first_category

[teststory:defaults]

category_list:Category 1,Category 2,Category 3
characters_list:Char A,Char B,Char C,Char D
ships_list:Char A/Char B,Char B & Char C,Char A & Char B,Char C/Char D,Char B/CharC

Result: (Calibre's tags showing prefix_category and prefix_ships as the entries, but I have first_category and first_ship in include_subject_tags)

Spoiler:

It looks like despite editing the string as a list the items still exist.

[julo]

Originally Posted by JimmXinu

Unless you can point to several stories like this, I'm going to declare this story a site bug that FFF isn't going to address.

Fair enough.

[JimmXinu]

Originally Posted by culytera

I'm trying to understand why my test entries aren't being used despite including only them as subject tags.

Basically what I'm trying to do is retrieving the first item in a list. I've managed to do this (maybe??) as shown on the title page, but for whatever reason it's taking other entries as subject tags. I didn't see any conflicting settings on overrides/defaults. Here's the relevant code:
...
It looks like despite editing the string as a list the items still exist.

You've exactly identified it. FFF is using the list, not the string.

The code for getting include_subject_tags is specifically calling for the lists (keeplists=True) which bypasses _LIST processing.

getSubjectTags() uses lists so it doesn't need to re-split things again. It doesn't matter for Calibre tags, but does very much for epub tags.

IE, if genre contains ['Action','Comedy'], you want to get:
<dc:subject>Action</dc:subject>
<dc:subject>Comedy</dc:subject>
Not:
<dc:subject>Action, Comedy</dc:subject>

That code--or rather its predecessor--almost certainly predates _LIST. And likely replace_metadata, etc. It's hard to track exact history past ~8 years ago due to the project name changing, code restructuring, and the project moving from code.google to github all at the same time.

So in all likelihood, I've just never considered this corner case before. And I'm not entirely sure how to address it.

• Just removing keeplists=True fixes your case, but leads directly to the <dc:subject> tag issue above.
• Re-splitting is complicated by the fact that each metadata entry list can configurably have a different separator (join_string_X setting). And include_in_X means it's not obvious which separator may apply.

Perhaps it's finally time to separate Calibre Tags from include_subject_tags... I'll need to think about this a bit.

BTW, kudos for figuring out teststory and using it well for its intended purpose.

[JimmXinu]

Originally Posted by JimmXinu

You've exactly identified it. FFF is using the list, not the string.

The code for getting include_subject_tags is specifically calling for the lists (keeplists=True) which bypasses _LIST processing.
...

After thinking about this for a while, I think the best solution may be to simply use the string version of each metadata entry in include_subject_tags and split by comma(,).

In some ways, this gives the user more control because they can use join_string_X to use a different separator than comma(,) to force an entry to be all one Tag, with a label even: <dc:subject>Genre: Action; Drama<dc:subject>

This does cause a change in behavior from the existing code: Value "A, B" will be come tags "A" and "B" instead of "A; B" in Calibre's tags.

In practice, I first expected this to be rare. But searching my own library found tags "House; M.D." and several variations on "Title; The" which were ',' on the site. ~100 books in my library of ~6200. With the proposed change, these would end up as tags "The" and "Title".

All of my examples come from tthfanfic.org, but how many other sites do something similar?

This could be handled with a couple replace_metadata lines. Or even a simple category,genre=>,=>; but I hesitate a lot to add add_to_replace_metadata lines to defaults.ini. And doing it in code would be worse.

So still thinking...

[riothamus]

Having some trouble with Scribblehub and Flaresolverr. I tried to update this story and just got timeout notices for the challenges. I updated Flaresolverr and still got the same timeouts. Any help would be appreciated.

Edited to remove NSFW link. My apologies.

[JimmXinu]

New Test Version Posted

2023-07-23
- Filter cookies for flaresolverr
- Add _LIST option to include_subject_tags(/extra_subject_tags), refactor

Originally Posted by JimmXinu

So still thinking...

So as usual, I decided changing existing behavior when it isn't very definitively broken is a Bad Thing.

Instead, I added a _LIST feature to include_subject_tags.

defaults.ini now says:

## include_subject_tags: entries to make epub subjects and calibre tags
## lastupdate creates two tags: "Last Update Year/Month: %Y/%m" and "Last Update: %Y/%m/%d"
##
## By default, entries listed in include_subject_tags are used as
## lists, skipping any <entry>_LIST processing in
## replace_metadata/etc.  You can instead use <entry>_LIST in
## include_subject_tags to get the whole list as one string.
##
## You can also use <entry>.SPLIT to split each value in the entry on
## ',' for individual values.  .SPLIT was basically obsoleted by \,
## splitting, but may still be used in some users' config.  Can also
## be combined with _LIST as <entry>_LIST.SPLIT to get both
## <entry>_LIST processing in replace_metadata/etc *and* split by ','
include_subject_tags: extratags, genre, category, characters, ships, status

For context, this .SPLIT ability was added in 2012 specifically for xenforo tagsfromtitles and \, splitting in 2014.

So with this test version, culytera just needs:

include_subject_tags:first_ship_LIST,first_category_LIST

I also refactored the getSubjectTags code. Which honestly worries me the most.

[JimmXinu]

Originally Posted by riothamus

Having some trouble with Scribblehub and Flaresolverr. I tried to update this story and just got timeout notices for the challenges. I updated Flaresolverr and still got the same timeouts. Any help would be appreciated.

I'm able to download the first few chapters of that with Flaresolverr v3.2.1. I would suspect you've tripped some additional blocking. Not much we can do to help with that.

BTW, I didn't look, but I'd guess that a story tagged 'Smut' is NSFW and probably shouldn't be direct linked here. (At least it wasn't naked pictures right on top like some story links.)

[riothamus]

Originally Posted by JimmXinu

I'm able to download the first few chapters of that with Flaresolverr v3.2.1. I would suspect you've tripped some additional blocking. Not much we can do to help with that.

BTW, I didn't look, but I'd guess that a story tagged 'Smut' is NSFW and probably shouldn't be direct linked here. (At least it wasn't naked pictures right on top like some story links.)

Oops, it's a story currently being published in book form, this is just the preview/editing version so I didn't even think about the NSFW rule. My apologies. I've edited the link out. I'll give it a try another day to download.

[booster-t]

Not sure if this is something with MY setup or a more general problem. As of today, the plugin gives me the file not found in cache error when it was working perfectly yesterday. The version on my desktop is Calibre 6.23 and fanficfare 4.25.10. But, if I go to my laptop, where the version is Calibre 6.15.1 and fanficfare is 4.22.1, the download works, and I have not changed anything. I'm on a Mac using Chrome if that makes a difference. The version of Chrome on my desktop is 115.0.5790.102 and cache is set to -1. Also, I turned off my VPN, and no difference.

An update

If I turn off use browser cache only and make open page in browser true, I get this message:

403 Client Error: Forbidden for url: https://www.fanfiction.net/s/1420059...e-Painted-Veil

update 2:

AO3 is fine ... no problem updating. Looks like its a FFN issue.

Stephen

[culytera]

Originally Posted by JimmXinu

BTW, kudos for figuring out teststory and using it well for its intended purpose.

Yes I didn't want to bombard AO3 with requests while testing my edits.

Originally Posted by JimmXinu

New Test Version Posted

2023-07-23
- Filter cookies for flaresolverr
- Add _LIST option to include_subject_tags(/extra_subject_tags), refactor

So with this test version, culytera just needs:

include_subject_tags:first_ship_LIST,first_category_LIST

Thank you so much! My test result looks like this now.

Now I'm testing lists where I want to only keep a certain item (in categories, not prefix_categories) when it's the only one that exists, while excluding it when a different entry exists.

For example, I want "Dragon Age" to be kept when *Inquisition/Origins/II aren't one of the entries as well. If one of those 3 exists, I want "Dragon Age" excluded from the list. I've tested something similar with Mass Effect and Mass Effect: Andromeda/Mass Effect Trilogy and for some reason, my regex worked with Mass Effect but not Dragon Age. Here are the tests (long image).

The code:

Spoiler:

[test1.com]
# URL: http://test1.com?sid=12345

add_to_extra_valid_entries:,prefix_category,prefix_ships,first_category,first_ship

include_in_prefix_ships:ships
include_in_prefix_category:category

include_in_first_ship:ships
include_in_first_category:category

keep_in_order_ships:true
keep_in_order_characters:true
keep_in_order_category:true
keep_in_order_prefix_ships:true
keep_in_order_prefix_category:true
keep_in_order_first_ship:true
keep_in_order_first_category:true

prefix_ships_label:Prefix Ships
prefix_category_label:Prefix Category
first_ship_label:First Ship
first_category_label:First Category

exclude_metadata_pre:
 prefix_ships=~(.*) & (.*)

replace_metadata:
 category=> (- All Media Types|- Fandom|\(Movies\)|\(Movie [0-9]+\)|\(Comics\)|\(TV\)|\(Video Games?\))$=>
 first_category_LIST=>(Dragon Age\,\s?)Dragon Age(:? (Inquisition|Origins|II))=>Dragon Age\2
 first_category_LIST=>(Mass Effect\,\s?)Mass Effect(:? (Andromeda|Trilogy))=>Mass Effect\2
 prefix_category=>^(.*)=>[c] \1
 prefix_ships=>^(.*)=>[p] \1
 first_category=>^(.*?)$=>[c] \1
 first_ship=>^(.*?\/.*?)$=>[p] \1
## first_category_LIST=>^(\[c\].*?)(\,.*)=>\1
 first_ship_LIST=>^(\[p\].*?\/.*?)(\,.*)=>\1
 first_category_LIST=>^(\[c\].*?)(\,.*)=>\1
exclude_metadata_post:
 category=~^(Dragon Age)$&&category_LIST=~Dragon Age:? (Inquisition|Origins|II)
 category=~^(Mass Effect)$&&category_LIST=~Mass Effect:? (Andromeda|Trilogy)

include_subject_tags:first_ship_LIST,first_category_LIST

titlepage_entries:<div class="inline">,seriesHTML,category,rating,warnings,</div>,ships,characters,prefix_ships,prefix_category,<div class="inline">,first_ship,first_category,</div>

titlepage_start:<?xml version="1.0" encoding="UTF-8"?>
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
 <html xmlns="http://www.w3.org/1999/xhtml">
 <head>
 <title>${title} by ${author}</title>
 <link href="stylesheet.css" type="text/css" charset="UTF-8" rel="stylesheet"/>
 <meta name="calibre-html-render-data" content='{"size": "A2", "margins": [0, 0, 0, 0]}'/>
 </head>
 <body class="fff_titlepage" style="font-size: 2.5vmax;">
 <div class="title">
 <h1><a href="${storyUrl}">${title}</a> by ${authorHTML}</h1>
 </div>
 <hr/>
 <dl class="tags">

titlepage_entry:
 <dt><b>${label}:</b></dt> <dd>${value}</dd>
titlepage_wide_entry:
wide_titlepage_entries:
titlepage_no_title_entry:
titlepage_end:
 <dl class="stats">
 </dl>
 </dl>
 <hr/>
 <div class="tag-summary-content">${description}</div>
 </body>
 </html>

[teststory:defaults]

category_list:Category 1,Category 2,Category 3
characters_list:Char A,Char B,Char C,Char D
ships_list:Char A/Char B,Char B & Char C,Char A & Char B,Char C/Char D,Char B/CharC
description:Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus faucibus massa vitae interdum maximus. Vestibulum dictum odio sed vestibulum vulputate. Proin placerat posuere ex, sit amet egestas ex tristique in. Quisque porta nisl et dui dignissim, sit amet rhoncus justo rhoncus. Proin sit amet leo a enim mollis molestie nec eget enim. Cras sit amet odio lorem. Ut ac fringilla nibh. Morbi faucibus sapien sit amet turpis venenatis, nec tincidunt risus cursus.

[teststory:10001]
title:{{storyId}} Dragon Age - All Media Types
category_list:Dragon Age - All Media Types
[teststory:10002]
title:{{storyId}} Dragon Age II,Dragon Age: Inquisition
category_list:Dragon Age II,Dragon Age: Inquisition
[teststory:10003]
title:{{storyId}} Dragon Age (Video Games),Dragon Age: Inquisition
category_list:Dragon Age (Video Games),Dragon Age: Inquisition
[teststory:10004]
title:{{storyId}} Dragon Age (Video Games),Dragon Age - All Media Types,Dragon Age: Origins,Dragon Age: Origins - Awakening
category_list:Dragon Age (Video Games),Dragon Age - All Media Types,Dragon Age: Origins,Dragon Age: Origins - Awakening
[teststory:20000]
title:{{storyId}} Mass Effect (Video Games)
category_list:Mass Effect (Video Games)
[teststory:20001]
title:{{storyId}} Mass Effect Trilogy
category_list:Mass Effect Trilogy
[teststory:20002]
title:{{storyId}} Mass Effect Trilogy,Mass Effect - All Media Types
category_list:Mass Effect Trilogy,Mass Effect - All Media Types
[teststory:20003]
title:{{storyId}} Mass Effect - All Media Types,Mass Effect: Andromeda
category_list:Mass Effect - All Media Types,Mass Effect: Andromeda
[teststory:20004]
title:{{storyId}} Mass Effect - All Media Types,Mass Effect: Andromeda,Mass Effect Trilogy
category_list:Mass Effect - All Media Types,Mass Effect: Andromeda,Mass Effect Trilogy

[JimmXinu]

Originally Posted by booster-t

Not sure if this is something with MY setup or a more general problem. As of today, the plugin gives me the file not found in cache error when it was working perfectly yesterday. The version on my desktop is Calibre 6.23 and fanficfare 4.25.10. But, if I go to my laptop, where the version is Calibre 6.15.1 and fanficfare is 4.22.1, the download works, and I have not changed anything. I'm on a Mac using Chrome if that makes a difference. The version of Chrome on my desktop is 115.0.5790.102 and cache is set to -1. Also, I turned off my VPN, and no difference.

Some people just have troubles sometimes. There's an FAQ about it.

[JimmXinu]

Originally Posted by culytera

For example, I want "Dragon Age" to be kept when *Inquisition/Origins/II aren't one of the entries as well. If one of those 3 exists, I want "Dragon Age" excluded from the list. I've tested something similar with Mass Effect and Mass Effect: Andromeda/Mass Effect Trilogy and for some reason, my regex worked with Mass Effect but not Dragon Age.

FYI, I personally consider trying to normalize AO3 tags a lost cause. I'll help with individual issues that can be articulated and tested, but I'm not interested in it in general.

In this case, you can't use use category_LIST in conditionals for category, that would recurse. The debug output will warn you about it:

Spoiler:

FFF: INFO: 2023-07-24 10:45:43,659: story.py(838): Skipping exclude_metadata_post key(category) value(Mass Effect) line(category=~^(Dragon Age)$&&category_LIST=~Dragon Age:? (Inquisition|Origins|II)) to prevent infinite recursion.
FFF: INFO: 2023-07-24 10:45:43,660: story.py(838): Skipping exclude_metadata_post key(category) value(Mass Effect) line(category=~^(Dragon Age)$&&category_LIST=~Dragon Age:? (Inquisition|Origins|II)) to prevent infinite recursion.
FFF: INFO: 2023-07-24 10:45:43,660: story.py(838): Skipping exclude_metadata_post key(category) value(Mass Effect) line(category=~^(Mass Effect)$&&category_LIST=~Mass Effect:? (Andromeda|Trilogy)) to prevent infinite recursion.
FFF: INFO: 2023-07-24 10:45:43,660: story.py(838): Skipping exclude_metadata_post key(category) value(Mass Effect: Andromeda) line(category=~^(Dragon Age)$&&category_LIST=~Dragon Age:? (Inquisition|Origins|II)) to prevent infinite recursion.
FFF: INFO: 2023-07-24 10:45:43,660: story.py(838): Skipping exclude_metadata_post key(category) value(Mass Effect: Andromeda) line(category=~^(Mass Effect)$&&category_LIST=~Mass Effect:? (Andromeda|Trilogy)) to prevent infinite recursion.
FFF: INFO: 2023-07-24 10:45:43,660: story.py(838): Skipping exclude_metadata_post key(category) value(Mass Effect Trilogy) line(category=~^(Dragon Age)$&&category_LIST=~Dragon Age:? (Inquisition|Origins|II)) to prevent infinite recursion.
FFF: INFO: 2023-07-24 10:45:43,660: story.py(838): Skipping exclude_metadata_post key(category) value(Mass Effect Trilogy) line(category=~^(Mass Effect)$&&category_LIST=~Mass Effect:? (Andromeda|Trilogy)) to prevent infinite recursion.
FFF: INFO: 2023-07-24 10:45:43,660: story.py(838): Skipping exclude_metadata_post key(category) value(Mass Effect: Andromeda) line(category=~^(Dragon Age)$&&category_LIST=~Dragon Age:? (Inquisition|Origins|II)) to prevent infinite recursion.
FFF: INFO: 2023-07-24 10:45:43,660: story.py(838): Skipping exclude_metadata_post key(category) value(Mass Effect Trilogy) line(category=~^(Dragon Age)$&&category_LIST=~Dragon Age:? (Inquisition|Origins|II)) to prevent infinite recursion.

[ownedbycats]

Originally Posted by culytera

For example, I want "Dragon Age" to be kept when *Inquisition/Origins/II aren't one of the entries as well. If one of those 3 exists, I want "Dragon Age" excluded from the list. I've tested something similar with Mass Effect and Mass Effect: Andromeda/Mass Effect Trilogy and for some reason, my regex worked with Mass Effect but not Dragon Age. Here are the tests (long image).

I'm a bit less familiar with Dragon Age, but if there's characters that only appear in certain games, something like this might also be useful for you. (ffnet and some pre-Andromeda AO3 stories have just a 'Mass Effect' category.)

replace_metadata: 
 category=>^Mass Effect$=>Mass Effect: Andromeda&&characters=~(Ryder|Jaal Ama Darav)
 category=>^Mass Effect$=>Mass Effect Trilogy&&characters=~(Shepard|Normandy Crew)

I don't think you can use a category conditional on a category replace though. I'm not sure why, perhaps to prevent accidental recursive loops.

[booster-t]

Originally Posted by JimmXinu

Some people just have troubles sometimes. There's an FAQ about it.

I followed the instructions — cleared the cache (actually deleted it) and now consistently get this response to any attempt to update:

HTTP Error in FFF '403 Client Error: Forbidden for url: https://www.fanfiction.net/s/1420059...-Veil'(403)

In the FWIW department, the line Browser_cache_path: ~/Library/Caches/Google/Chrome/Default/Cache/Cache_Data has the words Browser_cache_path in red

Stephen

[JimmXinu]

Originally Posted by booster-t

I followed the instructions — cleared the cache (actually deleted it) and now consistently get this response to any attempt to update:

HTTP Error in FFF '403 Client Error: Forbidden for url: https://www.fanfiction.net/s/1420059...-Veil'(403)

In the FWIW department, the line Browser_cache_path: ~/Library/Caches/Google/Chrome/Default/Cache/Cache_Data has the words Browser_cache_path in red

That suggests your setting isn't correct. For one thing, it's browser_cache_path--all lowercase.

FYI, if you set use_browser_cache_only:true for [www.fanfiction.net] it will only use cache.