[GUI Plugin] FanFicFare

[JimmXinu]

FYI, your posts were waiting for mod approval. Hopefully you have enough posts now to not have that problem anymore.

"how the old grace goes" is a substring of "care not how the old grace goes", but that shouldn't be a problem.

I'm not able to replicate the problem. I can have either of those two stories in library already and then download the other by URL with "Search by Title/Author(s)" on and they don't conflict.

When you have the "Search by Title/Author(s) for If Story Already Exists?" option checked FFF looks for existing books with the same title and author by using Calibre's db.find_identical_books() method(FFF src).

I don't know why you are seeing that. I can force it to happen if I use replace_metadata to force two different stories to have identical titles, but not just a substring. Or if I change the story name in library to match first.

db.find_identical_books() does 'fuzzy' title search, but unless there are Calibre settings for it or language setting differences, I don't know why we would get different results. Calibre code suggests the 'fuzzy' part refers to punctuation and whitespace differences.

If you run into it again, I would suggest one or more of:

1. Use the No then Yes buttons on the matching title/auth dialogs to download anyway. Then future updates will match by story URL.
2. Use 'Add Always' mode when downloading the second story.
3. Uncheck the "Search by Title/Author(s)" option
4. Use replace_metadata to change the title of one or the other, eg "how the old grace goes - Naruto".

[theoddpassenger]

I did unfortunately realize that only after submitting a comment from the full reply page and it told me it was waiting for moderator review

Anyway, thank you for the info! I tried deselecting 'Search by Title/Author(s) for If Story Already Exists?' and that did fix the issue I was having with those two particular fic.

One other unrelated question - is there/could there be a way to call a separate Generate Cover Plugin Setting specifically when downloading Anthologies? Right now I just re-generate the cover after downloading them which is not the end of the world but it would definitely be convenient if that was a configuration I could enable.

Thanks again!

[JimmXinu]

Originally Posted by theoddpassenger

One other unrelated question - is there/could there be a way to call a separate Generate Cover Plugin Setting specifically when downloading Anthologies? Right now I just re-generate the cover after downloading them which is not the end of the world but it would definitely be convenient if that was a configuration I could enable.

There's a section on anthology covers in the wiki page.

That section may help, but it assumes you already know the terminology defined at the beginning of the page.

[legacyx3]

Hey, I noticed lately that whenever I try to get URLs from a Wattpad reading list it will stop at 32. This happens even when there are more than 32 URLs in the reading list that I do not have downloaded. Is there a setting I accidentally enabled? I thought I was about to get more than 32 URLs before.

[JimmXinu]

It looks like wattpad only includes 32 list items in the initial HTML page now and fetches more stories using additional requests from javascript, which it gets back as JSON. FFF can't run javascript.

The easiest solution is to instead open FFF's 'Download from URLs' dialog (or 'Make Anthology Epub from URLs'), then go to the list page in your browser, select all text in the page, then copy/paste or drag/drop the text into the FFF dialog.

FFF will parse the HTML from the clipboard as seen in the browser to get the story URLs.

[Wolgan]

The scenario is that I have applied the content rating normalization suggested here. However, fictionmania.tv has a special site rating "XXX-I" to warn of stories with specific activities. I definitely want to capture that in the warnings, and have come up with the following arrangement.

If I run it without the add_to_replace_metadata, it works correctly, and applies the warning, but leaves the rating unchanged. If I include the add_to_replace_metadata line, it normalizes the rating, but doesn't apply the warning.

I have a large personal.ini , so I removed everything but the code block below, and the behavior doesn't change, so I don't believe there should be anything conflicting with these.

[fictionmania.tv]

add_to_extra_valid_entries:warnings

extrawarnings:CAUTION

exclude_metadata_pre:
 warnings==CAUTION&&rating!=XXX-I

add_to_replace_metadata:
 rating=>XXX-I=>NC-17

Two questions; 1) is there a more efficient/elegant way to accomplish what I'm trying to do? 2) Am I misunderstanding the _pre part of the exclude_metadata_pre, or is this a bug, edge case, or just syntax error on my part somewhere?


EDIT: I changed the line to the following (aka reversed the logic), and it works now, however I'm still unsure what I did wrong in my original example.

include_metadata_pre:
 warnings==CAUTION&&rating==XXX-I

Extra EDIT: Still broken, now it applies to warning to all stories.

[JimmXinu]

The conditional part of a in/exclude_metadata_pre line (after &&) uses the same routines to get metadata values as everything else. Which means replace_metadata has already been applied to rating when the warnings _metadata_pre line is applied.

The wiki page mentions that for replace_metadata but not for in/exclude specifically:

NOTE: The way it works is that include/exclude_pre then replace_metadata then include/exclude_post are applied to each metadata entry as it's used, not all metadata as a whole. When using replace_metadata conditional clauses (see ReplaceMetadata), be aware that the include/exclude_metadata for both pre and post will already be applied to the metadata entries you are using in the conditional.

I will update that wording. Changing the behavior is a huge can of worms I'm not opening.

FYI, warnings is a standard metadata entry, you don't need to use add_to_extra_valid_entries:warnings. And be aware that extrawarnings is not universal, there are only a handful of standard metadata entries that have extraX settings.

Here's a solution that works for me:

add_to_extra_valid_entries:rating_raw
include_in_rating_raw:rating.NOREPL
include_in_warnings:warnings,"CAUTION"

exclude_metadata_pre:
 warnings==CAUTION&&rating_raw!=XXX-I

add_to_replace_metadata:
 rating=>^XXX-I$=>NC-17

I've created a raw copy of rating, replaced the extrawarnings line, and matched only "XXX-I" as an entire string. So if "XXX-II" shows up, it doesn't give you "NC-17I".

[ownedbycats]

Would it be technically possible and/or make sense to have conditionals accept a NOREPL directly? Example:

exclude_metadata_pre:
 warnings==CAUTION&&rating.NOREPL!=XXX-I

[JimmXinu]

I know I've considered it before. I don't remember off hand why I decided not to. The whole metadata processing system is pretty fragile. As is, .NOREPL is implemented cleanly in one place.

[Tova]

Hi, first of thanks for the awesome plugin I love it! But I have one problem I can't get to solve on my own.
I have a pocketbook on which I'd like to sort all my ebooks under the internal "genre" sorting showing only the pairings from fanfics downloaded from ao3. I searched on the forum for solutions but only got it to work semi ok, with now only the character tags and relationships tag but also still some other tags showing up (like fandoms, status of completion etc.). Is it possible for it to only show the pairings tag under "genre" on pocketbook?
I have added my ini below, any help would be much appreciated!

## This is an example of what your personal configuration might look
## like. Uncomment options by removing the '#' in front of them.

[defaults]
## [defaults] section applies to all formats and sites but may be
## overridden at several levels. See
## https://github.com/JimmXinu/FanFicFare/wiki/INI-File for more
## details.

## Some sites also require the user to confirm they are adult for
## adult content. Uncomment by removing '#' in front of is_adult.
#is_adult:true

## Don't like the numbers at the start of chapter titles on some
## sites? You can use strip_chapter_numbers to strip them off. Just
## want to make them all look the same? Strip them off, then add them
## back on with add_chapter_numbers. Don't like the way it strips
## numbers or adds them back? See chapter_title_strip_pattern and
## chapter_title_add_pattern in defaults.ini.
#strip_chapter_numbers:true
#add_chapter_numbers:true

exclude_notes:authorheadnotes,chapterheadnotes,cha pterfootnotes,authorfootnotes


[archiveofourown.org]
use_basic_cache:true
## Some sites require login (or login for some rated stories) The
## program can prompt you, or you can save it in config. In
## commandline version, this should go in your personal.ini, not
## defaults.ini.
username:XXXXXXXX
password:XXXXXXXX


is_adult:true
always_login:true

## Some adapters collect additional meta information beyond the
## standard ones. They need to be defined in extra_valid_entries to
## tell the rest of the FanFicFare system about them. They can be
## used in include_subject_tags, titlepage_entries,
## extra_titlepage_entries, logpage_entries, extra_logpage_entries,
## and include_in_* config items. You can also add additional entries
## here to build up composite metadata entries.
## archiveofourown.org, for example, fills genre (a standard
## entry) as the composite of freeformtags, ao3categories in
## include_in_genre. If there's ever more than 4 series, add
## series04,series04Url etc.
extra_valid_entries:fandoms,ao3categories, comments, chapterslashtotal, chapterstotal, kudos,
hits, bookmarks, collections, byline, bookmarked, bookmarktags,
bookmarksummary, bookmarkprivate, bookmarkrec, restricted, series00,
series01, series02, series03, series00Url, series01Url, series02Url,
series03Url, series00HTML, series01HTML, series02HTML, series03HTML,
ships2chars,onlyromships
fandoms_label:Fandoms
freeformtags_label:Relationsips
freefromtags_label:Relationsips
ao3categories_label:Category
comments_label:Comments
chapterslashtotal_label:Chapters/Total Chapters
chapterstotal_label:Total Chapters
kudos_label:Kudos
hits_label:Hits
collections_label:Collections
## Count of bookmarks on story by all users
bookmarks_label:Bookmarks
## Tags & Summary from *your* bookmark on the story. Only collected
## if always_login:true
bookmarked_label:I Bookmarked Story
bookmarktags_label:My Bookmark Tags
bookmarksummary_label:My Bookmark Summary
bookmarkprivate_label:My Bookmark Private
bookmarkrec_label:My Bookmark Rec
restricted_label:Restricted to Registered Users
series00HTML_label:Series
series01HTML_label:Additional Series
series02HTML_label:Additional Series
series03HTML_label:Additional Series

## Assume entryUrl, apply to "<a class='%slink' href='%s'>%s</a>" to
## make entryHTML.
make_linkhtml_entries:series00,series01,series02,s eries03

## chapterslashtotal contains 1/3 or 1/1 or 3/? etc as reported by AO3.
## chapterstotal is just the total chapters part after the /
include_in_chapterstotal:chapterslashtotal.NOREPL
add_to_replace_metadata:
chapterstotal=>^([0-9])+/(.+)$=>\2

keep_in_order_freeformtags:true
keep_in_order_ao3categories:true
keep_in_order_fandoms:true
keep_in_order_genre:true
keep_in_order_category:true
keep_in_order_characters:true
keep_in_order_ships:true

## AO3 doesn't have anything it calls 'genre'. The adapter used to be
## hardcoded to include the site specific metadata freeformtags &
## ao3categories in the standard metadata field genre. By making it
## configurable, users can change it.
include_in_genrenlyromships
exclude_in_genre:freeformtags

## AO3 uses the word 'category' differently than most sites. The
## adapter used to be hardcoded to include the site specific metadata
## fandom in the standard metadata field category. By making it
## configurable, users can change it.
include_in_category:fandoms

## freeformtags was previously typo'ed as freefromtags. This way,
## freefromtags will still work for people who've used it.
#include_in_freefromtags:freeformtags

## adds to titlepage_entries instead of replacing it.
#extra_titlepage_entries: fandoms,freeformtags,ao3categories,comments,
#chapterslashtotal,kudos,hits,bookmarks,bookmarked ,bookmarktags,
#bookmarksummary,series01HTML,series02HTML,series0 3HTML
titlepage_entries: seriesHTML,rating,warnings,ao3categories,fandoms,
ships,status,datePublished,dateUpdated,
numWords,chapterslashtotal,bookmarked,
bookmarktags,bookmarksummary,series01HTML,series02 HTML,series03HTML,
site,description

## adds to include_subject_tags instead of replacing it.
extra_subject_tags:fandoms,onlyromships

include_in_onlyromships:ships

add_to_include_metadata_post:
onlyromships=~[ ]*/[ ]*

#add_to_exclude_metadata_post:
#onlyromships=~&amp;

#include_in_ships2chars:ships.NOREPL
# NOREPL prevents any ships patterns from being applied to ships2chars.

#add_to_replace_metadata:
# ships2chars=>[ ]*/[ ]*=>/
# ships2chars=>[ ]*&amp;[ ]*=>/
# ships2chars=>([^/]+)/([^/]+)/([^/]+)/(.+)=>\1\,\2\,\3\,\4
# ships2chars=>([^/]+)/([^/]+)/(.+)=>\1\,\2\,\3
# ships2chars=>([^/]+)/(.+)=>\1\,\2

include_in_characters:characters,ships2chars

add_to_custom_columns_settings:
chapterslashtotal=>#chapters
freeformtags=>#freeformtags
onlyromships=>#ships
ships=>#gcships
description=>#description
bookmarked=>#bookmarked

extra_logpage_entries:fandoms,freeformtags,ao3cate gories,comments,
chapterslashtotal,kudos,hits,bookmarks,collections ,byline,bookmarked,
bookmarktags,bookmarksummary,bookmarkprivate,bookm arkrec,restricted,series00,
series01,series02,series03


[epub]
## Include images from img tags in the body and summary of stories.
## Images will be converted to jpg for size if possible. Images work
## in epub format only. To get mobi or other format with images,
## download as epub and use Calibre to convert.
## true by default, uncomment and set false to not include images.
#include_images:true

## If set false, the summary will have all html stripped for safety.
## Both this and include_images must be true to get images in the
## summary.
## true by default, uncomment and set false to not keep summary html.
#keep_summary_html:true

## If set true, and there isn't a specific cover image, the first
## image found in the story will be made the cover image. If
## keep_summary_html is true, images in the summary will be before any
## in chapters.
## true by default, uncomment and set false to turn off
#make_firstimage_cover:true


## Most common, I expect will be using this to save username/passwords
## for different sites. Here are a few examples. See defaults.ini
## for the full list.

[www.twilighted.net]
#username:XXXXXXXX
#password:XXXXXXXX
## default is false
#collect_series: true

[www.fimfiction.net]
#is_adult:true
#fail_on_password:XXXXXXXX

[www.tthfanfic.org]
#is_adult:true
## tth is a little unusual--it doesn't require user/pass, but the site
## keeps track of which chapters you've read and won't send another
## update until it thinks you're up to date. If you set
## username/password, FFF will login to download. Then the site
## thinks you're up to date.
#username:XXXXXXXX
#password:XXXXXXXX


## This section will override anything in the system defaults or other
## sections here.
[overrides]
## default varies by site. Set true here to force all sites to
## collect series.
#collect_series: true

[JimmXinu]

Tova,

A few things unrelated to your question:

1. I've approved your post--new users need their posts approved for the first 10 or so--you don't need to try again, just be patient.
2. It works better to wrap ini sections in [code] instead of [quote] to preserve spacing etc.
3. Long ini [code] sections should be further wrapped in [spoiler] tags.

Re: your question - I don't know anything about 'pocketbook'. Where does it get it's 'internal genre' from?

[Wolgan]

Originally Posted by JimmXinu

The conditional part of a in/exclude_metadata_pre line (after &&) uses the same routines to get metadata values as everything else. Which means replace_metadata has already been applied to rating when the warnings _metadata_pre line is applied.

The wiki page mentions that for replace_metadata but not for in/exclude specifically:

I will update that wording. Changing the behavior is a huge can of worms I'm not opening.

FYI, warnings is a standard metadata entry, you don't need to use add_to_extra_valid_entries:warnings. And be aware that extrawarnings is not universal, there are only a handful of standard metadata entries that have extraX settings.

Here's a solution that works for me:

add_to_extra_valid_entries:rating_raw
include_in_rating_raw:rating.NOREPL
include_in_warnings:warnings,"CAUTION"

exclude_metadata_pre:
 warnings==CAUTION&&rating_raw!=XXX-I

add_to_replace_metadata:
 rating=>^XXX-I$=>NC-17

I've created a raw copy of rating, replaced the extrawarnings line, and matched only "XXX-I" as an entire string. So if "XXX-II" shows up, it doesn't give you "NC-17I".

That makes perfect sense, thank you for explaining!

[Tova]

[QUOTE=JimmXinu;4355013]Tova,

A few things unrelated to your question:

1. I've approved your post--new users need their posts approved for the first 10 or so--you don't need to try again, just be patient.
2. It works better to wrap ini sections in [code] instead of

   to preserve spacing etc.

   Long ini [code] sections should be further wrapped in [spoiler] tags.

Re: your question - I don't know anything about 'pocketbook'. Where does it get it's 'internal genre' from?

Oh sorry my bad I didn't know, will keep in mind for future posts =)

I think it takes everything that's in the "freeformtags" it imports from fanficfare, I tried around a bit and got it reduced to the tags from my original post (characters + pairings + some other random tags) but I can't figure out how to keep only the pairings for sorting.

[JimmXinu]

Originally Posted by Tova

I think it takes everything that's in the "freeformtags" it imports from fanficfare, I tried around a bit and got it reduced to the tags from my original post (characters + pairings + some other random tags) but I can't figure out how to keep only the pairings for sorting.

I would guess that it's coming from subject tags, which also controls what goes in Calibre's Tags column.

Default is:

Spoiler:

## include_subject_tags: entries to make epub subjects and calibre tags
## lastupdate creates two tags: "Last Update Year/Month: %Y/%m" and "Last Update: %Y/%m/%d"
##
## By default, entries listed in include_subject_tags are used as
## lists, skipping any <entry>_LIST processing in
## replace_metadata/etc.  You can instead use <entry>_LIST in
## include_subject_tags to get the whole list as one string.
##
## You can also use <entry>.SPLIT to split each value in the entry on
## ',' for individual values.  .SPLIT was basically obsoleted by \,
## splitting, but may still be used in some users' config.  Can also
## be combined with _LIST as <entry>_LIST.SPLIT to get both
## <entry>_LIST processing in replace_metadata/etc *and* split by ','
include_subject_tags: extratags, genre, category, characters, ships, status

[hayley114]

Hello,

I have updated my personal.ini with my AO3 username, password, and always_login:true. I did this by copying the whole AO3 section from default.ini into my personal.ini, perhaps that's where I went wrong? Something is not working because my private bookmarks and works that authors have restricted to logged-in users only are not showing up in calibre after I get story URLs from webpage. I'm sorry if I'm missing something obvious; I'm very new to all of this.

This plugin is amazing and such a lifesaver! Thank you so much for making it!!!

Best,

Hayley