[GUI Plugin] FanFicFare

[junni]

Hi

I'm trying to create an adapter for novelfullplus.com which is very similar to readnovelfull.com, but since the site does not conform with HTML for novel links, example:https://novelfullplus.com/novel/aban...an-kings-mate/ , i keep getting that novel link is not a valid story URL. Can you help me Jim?

[JimmXinu]

Originally Posted by junni

HiI'm trying to create an adapter for novelfullplus.com which is very similar to readnovelfull.com, but since the site does not conform with HTML for novel links, example:https://novelfullplus.com/novel/aban...an-kings-mate/ , i keep getting that novel link is not a valid story URL. Can you help me Jim?

I don't understand what you mean by "the site does not conform with HTML for novel links". I don't see anything that doesn't conform with HTML in that link.

If you mean that you are trying to copy adapter_novelfull.py and you're having trouble with it recognizing the story URLs, look at the getSiteURLPattern() function. I assume you already figured out you need to change getSiteDomain().

[junni]

Originally Posted by JimmXinu

I don't understand what you mean by "the site does not conform with HTML for novel links". I don't see anything that doesn't conform with HTML in that link.

If you mean that you are trying to copy adapter_novelfull.py and you're having trouble with it recognizing the story URLs, look at the getSiteURLPattern() function. I assume you already figured out you need to change getSiteDomain().

Yeah, i have to change the regular expression, but i don't seem to find the right one for this kind of link:

https://novelfullplus.com/novel/aban...can-kings-mate

[JimmXinu]

It should be very straight forward if you understand regular expressions.

I don't know that site, but if you assume that you don't need the optional index.php and the trailing .htm(l) novelfull.com uses:

    def getSiteURLPattern(self):
        return r"https?://%s/novel/(?P<story_id>[^/]+)(/.*)?" % re.escape(self.getSiteDomain())

Note that you will need change the self._setURL() in __init__() too.

[junni]

Originally Posted by JimmXinu

It should be very straight forward if you understand regular expressions.

I don't know that site, but if you assume that you don't need the optional index.php and the trailing .htm(l) novelfull.com uses:

    def getSiteURLPattern(self):
        return r"https?://%s/novel/(?P<story_id>[^/]+)(/.*)?" % re.escape(self.getSiteDomain())

Note that you will need change the self._setURL() in __init__() too.

Thanks Jim, got it to work thanks to you. Regular expressions are so confusing

[ownedbycats]

Originally Posted by junni

Thanks Jim, got it to work thanks to you. Regular expressions are so confusing

Regex101 has helped me. Make sure it's using Python as other languages have subtle differences that might muck things up.

[ownedbycats]

PSA: Flaresolverr doesn't seem to be working at the moment, see here. If you have issues, use browser cache for now.

EDIT: Just curious: would a 'retry last download' button in the FFF dropdown be viable? There are a few use cases I can see for it; switching from Flaresolverr to cache, or changing a replace_metadata.

[JimmXinu]

Originally Posted by ownedbycats

EDIT: Just curious: would a 'retry last download' button in the FFF dropdown be viable? There are a few use cases I can see for it; switching from Flaresolverr to cache, or changing a replace_metadata.

It's theoretically feasible. But with all the different update modes and options it's not necessary straightforward to use or implement.

Plus, my first thought is users hammering it when they got throttled/blocked and only making it worse.

But I'll think about it more if there are more users interested.

[Ser4nb2LUY6e]

Any chance to get support for

https://hostednovel.com/

added?

As far as I can tell, it's just plain html pages and doesn't require any java script for basic function (I can access chapter lists and chapter contents even with everything blocked).

[Ser4nb2LUY6e]

Originally Posted by ownedbycats

PSA: Flaresolverr doesn't seem to be working at the moment, see here. If you have issues, use browser cache for now.

EDIT: Just curious: would a 'retry last download' button in the FFF dropdown be viable? There are a few use cases I can see for it; switching from Flaresolverr to cache, or changing a replace_metadata.

I'm still running 2.2.4 and it seems to work fine for me for any site that I use flaresolverr for...

[AbyssalAriel]

Hello, I have a few questions.

1. Downloading from AO3 or other places, can I have each of the tags keep their link/HTML in the titlepage? When downloading from AO3 directly, their epub has links on each of the tags (as well as the series, and any collections), but FFF doesn't (except for series apparently).

If that's not possible, is there at least a way to make the titlepage underline each of the tags/entries? They're separated by comma, but it would be a lot more readable if each individual entry was underlined (so the lack of underlining between them breaks them up more).

2. How can I make a single metadata entry that's composed of others, plus some hardcoded text? I want to have the publish, update, wordcount and chapter count all on one line in the titlepage, and the only way I can think of to do this make an extra field and import the data inside, something like:

customfieldname: Published: {datePublished} | Updated {dateUpdated} | Chapters: {chapterslashtotal}

Unsurprisingly, that syntax doesn't work. But is there a way to achieve the same end goal, and what is it?

3. If accomplishing what I want in question 2, can I conditionally modify it? For example, I'd want to exclude/remove the "Updated" section if it's the same as the publish date.

4. How can I replicate `Status:Completed` for a custom column, inside personal.ini? I want to have a "Completed" custom column that tracks if a work is complete, but it doesn't seem to behave with sites other than AO3 (I assume because they don't have a `status` entry at all), so I want only AO3 to update that column (for the other sites I'll just enter it manually, but I don't want the value to be overwritten if I update from metadata again).

5. Can I somehow update/rebuild the epub titlepage, using the Saved Metadata instead of redownloading the whole work?

So that I can do things like, for example, mess with the titlepage_entries (adding or removing or re-ordering them), or normalize genre tags, without hitting the site servers again when I just want to change the titlepage, and in case it's deleted.

In case it helps at all, here is my personal.ini:

Spoiler:

[defaults]

is_adult:true

#################################################
#################################################

[epub]
extratags:

output_css:
 body { background-color: #%(background_color)s;
        text-align: left;
        margin: 2%%;
	adobe-hyphenate: none; }
 pre { font-size: x-small; }
 h1 { text-align: center; }
 h2 { text-align: center; }
 h3 { text-align: center; }
 h4 { text-align: center; }
 h5 { text-align: center; }
 h6 { text-align: center; }
 .label_value {
    display: block;
    margin: 0 0 1em 1em
    }
 .CI {
     text-align:center;
     margin-top:0px;
     margin-bottom:0px;
     padding:0px;
     }
 .center   {text-align: center;}
 .cover    {text-align: center;}
 .full     {width: 100%%; }
 .quarter  {width: 25%%; }
 .smcap    {font-variant: small-caps;}
 .u        {text-decoration: underline;}
 .bold     {font-weight: bold;}
 .big { font-size: larger; }
 .small { font-size: smaller; }

## To remove summary/description
wide_titlepage_entries: storyUrl, authorUrl, seriesUrl

## ${id}, ${label}, ${value}
titlepage_entry:<b>${label}:</b><br />
 <dd class="label_value">${value}</dd>

#################################################
#################################################

[archiveofourown.org]

add_to_extra_valid_entries:,romships,platships
romships_label:Romantic Ships
platships_label:Platonic Ships

include_in_romships:ships
include_in_platships:ships

# only include / ships in romships and & ships in platships.
# &amp; because HTML
add_to_include_metadata_post:
 romships=~/
 platships=~&amp;

keep_in_order_freeformtags:true
keep_in_order_ao3categories:true
keep_in_order_fandoms:true
keep_in_order_genre:true
keep_in_order_category:true
keep_in_order_characters:true
keep_in_order_ships:true
keep_in_order_romships:true
keep_in_order_platships:true

add_to_replace_metadata:
 publisher=>archiveofourown.org=>Archive of Our Own

exclude_metadata_pre:
 freeformtags,genre==Other Additional Tags to Be Added

titlepage_entries:rating,warnings,ao3categories,fandoms,
 romships,platships,characters,genre,
 bookmarked,bookmarktags,bookmarksummary,
 seriesHTML,series01HTML,series02HTML,series03HTML,collections,
 datePublished,dateUpdated,numWords,chapterslashtotal,
 description

## AO3 doesn't have anything it calls 'genre'.
include_in_genre:freeformtags
## Only includes the normal tags.
include_subject_tags:freeformtags
custom_columns_settings:
 ao3categories=>#ao3_category
 romships=>#ships
 platships=>#genships

strip_chapter_numbers:true
add_chapter_numbers:toconly
chapter_title_strip_pattern:^([0-9]+[\.: -]+)?(Chapter *[0-9]+[\.:, -]*)?

[Chrystalline]

Is there an age issue with FFN stories now? I switched to using browser cache when their latest update broke FFDL (and therefore FFF couldn't use its cache) but I have a story that updated today (like an hour ago) and so I went through it in my browser and told FFF to update it in Calibre, and it's acting like there is no update to add. It says it's a good URL (which it wouldn't if I'd failed to go through it in the browser first) but it only updates the Checked Date and nothing else. Do I need to wait a longer time after FFN notifies me of the update before I try this?

https://www.fanfiction.net/s/1239721...in-Reformation

ETA: It looks like chapter 1 is still showing the old metadata, while the later chapters have the updated metadata, including the Updated Date and chapter count. I'm guessing this is an FFN glitch and nobody knows when it'll fix.
Yep, chapter 1 finally updated properly on FFN, and the update in Calibre went fine. I guess I just have to leave it for at least a couple hours to make sure FFN gets its metadata straight before updating.

[JimmXinu]

Originally Posted by Chrystalline

Is there an age issue with FFN stories now?
...

It's pretty common for ffnet to not update the metadata on the first chapter immediately. FFF has an option (check_next_chapter) to look for another chapter past the one reported by chapter one.

It's off by default to avoid extra network hits, but if you're using browser cache only, it doesn't matter. This is what I use these days:

[www.fanfiction.net]
use_browser_cache:true
use_browser_cache_only:true
never_make_cover:true
check_next_chapter:true

[JimmXinu]

Originally Posted by AbyssalAriel

1. Downloading from AO3 or other places, can I have each of the tags keep their link/HTML in the titlepage? When downloading from AO3 directly, their epub has links on each of the tags (as well as the series, and any collections), but FFF doesn't (except for series apparently).

You are correct, FFF doesn't currently do that. It could (probably) be done, but at the expense of not being able to order the entries in the list.

Given the absolute mess that AO3 tagging is, I'm not very inclined to pursue this. Especially since it would only save one click--click the story link, then you have all the tags links right there.

Originally Posted by AbyssalAriel

If that's not possible, is there at least a way to make the titlepage underline each of the tags/entries? They're separated by comma, but it would be a lot more readable if each individual entry was underlined (so the lack of underlining between them breaks them up more).

You can customize the titlepage in a number of ways, there's a wiki page.

However, to apply an underline to each entry in a list, but not the whole list, I think you'd have to wrap tags around each entry using replace_metadata. There's a wiki page for that, too.

I'd copy the list, change that copy, then include it in the titlepage rather than change the original list if you don't also want that HTML showing up in Calibre tags, etc. More about add_to_extra_valid_entries below.

Originally Posted by AbyssalAriel

2. How can I make a single metadata entry that's composed of others, plus some hardcoded text? I want to have the publish, update, wordcount and chapter count all on one line in the titlepage, and the only way I can think of to do this make an extra field and import the data inside, something like:

customfieldname: Published: {datePublished} | Updated {dateUpdated} | Chapters: {chapterslashtotal}

Unsurprisingly, that syntax doesn't work. But is there a way to achieve the same end goal, and what is it?

The only solution that comes to mind is to:

1. Use add_to_extra_valid_entries and include_in_* to create copies of datePublished, dateUpdated and chapterslashtotal so you aren't affecting default processing that use them, plus your new entry, let's call it mergedentry.
2. Use replace_metadata to add the label to each of those copied entries.
3. Use include_in_mergedentry to put them in one list and keep_in_order_mergedentry:true to suppress list sorting.
4. Use join_string_mergedentry:\s|\s to change the separator.

Originally Posted by AbyssalAriel

3. If accomplishing what I want in question 2, can I conditionally modify it? For example, I'd want to exclude/remove the "Updated" section if it's the same as the publish date.

Should be possible using appropriate regexps in replace_metadata on mergedentry_LIST. There was a discussion earlier this month on something similar.

Originally Posted by AbyssalAriel

4. How can I replicate `Status:Completed` for a custom column, inside personal.ini? I want to have a "Completed" custom column that tracks if a work is complete, but it doesn't seem to behave with sites other than AO3 (I assume because they don't have a `status` entry at all), so I want only AO3 to update that column (for the other sites I'll just enter it manually, but I don't want the value to be overwritten if I update from metadata again).

Using add_to_custom_columns_settings under [archiveofourown.org] for that column will only set it for AO3 stories.

FFF has no way of knowing when you have edited a Calibre custom column vs FFF previously setting it. If you don't want FFF to ever overwrite your changes, the only way with custom_columns_settings is to use the 'n' mode option so it is only used with new books. See the comments in defaults.ini for custom_columns_settings.

Originally Posted by AbyssalAriel

5. Can I somehow update/rebuild the epub titlepage, using the Saved Metadata instead of redownloading the whole work?

No. It's been requested and considered more than once and it just doesn't fit into the plugin.

For repetitive testing, I use FFF's CLI with the --save-cache option to cache the pages once and use them over and over. However, officially, that option is undocumented & unsupported.

[ownedbycats]

Originally Posted by Chrystalline

Is there an age issue with FFN stories now? I switched to using browser cache when their latest update broke FFDL (and therefore FFF couldn't use its cache) but I have a story that updated today (like an hour ago) and so I went through it in my browser and told FFF to update it in Calibre, and it's acting like there is no update to add. It says it's a good URL (which it wouldn't if I'd failed to go through it in the browser first) but it only updates the Checked Date and nothing else. Do I need to wait a longer time after FFN notifies me of the update before I try this?

https://www.fanfiction.net/s/1239721...in-Reformation

ETA: It looks like chapter 1 is still showing the old metadata, while the later chapters have the updated metadata, including the Updated Date and chapter count. I'm guessing this is an FFN glitch and nobody knows when it'll fix.
Yep, chapter 1 finally updated properly on FFN, and the update in Calibre went fine. I guess I just have to leave it for at least a couple hours to make sure FFN gets its metadata straight before updating.

For the future, check_next_chapter:true under the ffnet section of your personal.ini can help with this.