[GUI Plugin] FanFicFare

[JimmXinu]

New Test Version Posted

Changes to tagsfromtitle needed a fix to avoid tags with ';' in them instead of individual tags.

2019-05-08
- Fix for base_xenforo's tagsfromtitle needing to be split.
- base_xenforoforum: include forumtags in genre and tagsfromtitle in category instead of including both in subject_tags.
- Don't hardcode extratags into subject tags--it's in include_subject_tags in defaults.ini.
- Refactor base_xenforoforum for upcoming XenForo2 update for SV.

[Tanjamuse]

If I copy the following into the metadata-column can you add a tweak or something that would let Calibre sort it out on its own?

Spoiler:

The One Where Murray is Accused of Murdering a Client, and Nick and Cody Don’t Know if He Did It

by der_tanzer



Category: Riptide (TV)
Genre: M/M
Language: English
Status: Completed
Published: 2011-12-16
Updated: 2011-12-16
Packaged: 2017-10-27 10:13:32
Rating: Teen And Up Audiences
Warnings: No Archive Warnings Apply
Chapters: 1
Words: 1,247
Publisher: archiveofourown.org
Story URL: https://archiveofourown.org/works/294608
Author URL: https://archiveofourown.org/users/de...uds/der_tanzer
Summary: <blockquote class="userstuff">
<p>Sometimes even the people who know you best have doubts.</p>
</blockquote>

[ILB]

Originally Posted by JimmXinu

I've been working on implementing Title Page update as a separate feature, but it's not going to do what many users will expect.

Specifically, it occurred to me that the first thing a new user is likely to try with it is changing the Title in Calibre, doing 'Refresh Title Page' and then not understanding why their changed title doesn't appear in the title page.

And I think maybe most don´t know there is a difference between cover and title page.
If I wouldn´t use both in different ways I probably would not care much about that, too.

Originally Posted by JimmXinu

I'm starting to feel the 'correct' answer, if you want to put Calibre metadata/columns into your ebooks, is to use Calibre's built-in metadata jacket.

Admittedly, I haven't explored the idea any further than, "yes, metadata jacket is customizable", but it offers some clear advantages--most prominently that can be used on ebooks from other sources and other formats(than epub).

It´s as baggins41 said: I want to avoid re-downloading the book I already have (to avoid unnecessary traffic on the site) or in case the original story no longer exists because than it gets an error and no updated title page.
If there is another way to “update” title page without downloading the book again from the site that would be enough for me.

But can I use Calibre's built-in metadata jacket to add custom columns (which I maintain manually after downloading) to my title page?
And how do I update my book to do this?

[JimmXinu]

Originally Posted by Tanjamuse

If I copy the following into the metadata-column can you add a tweak or something that would let Calibre sort it out on its own?

That doesn't change the issues previously mentioned, it just moves them to a different place.

Originally Posted by ILB

It´s as baggins41 said: I want to avoid re-downloading the book I already have (to avoid unnecessary traffic on the site) or in case the original story no longer exists because than it gets an error and no updated title page.
If there is another way to “update” title page without downloading the book again from the site that would be enough for me.

But can I use Calibre's built-in metadata jacket to add custom columns (which I maintain manually after downloading) to my title page?
And how do I update my book to do this?

Calibre's Polish feature has an option for add/update (and one for remove) 'metadata jacket'. The jacket is placed directly after the cover (if present). It does not effect the FFF title page, nor does it appear in the TOC like FFF title page does in multi-chapter books.

You can customize it by copying the default jacket template directory to your personal resources dir and then editing the files. The manual page explains it, but it may be a little out of date as for me the source dir is C:\Program Files\Calibre2\app\resources\jacket . Custom columns can be added by using _ instead of #. For #genre:

        <div>
          <p class="cbj_label">{_genre_label}:</p>
          <p class="cbj_content">{_genre}</p>
        </div>

FFF on update will discard the jacket file. But FFF already has an option to invoke the Polish option to smarten punctuation and I would be willing to add a similar option to automatically call Polish update jacket.

UPDATE: Calling Polish to update jacket might be more difficult than that--there are some 'interesting' interactions between conversion and jacket I would need to explore more.

(BTW, note that if you use Polish to add/update cover page, it will name the cover page file titlepage.xhtml. FFF's title page (title_page.xhtml) is unchanged.)

[ilovejedd]

Originally Posted by ILB

But can I use Calibre's built-in metadata jacket to add custom columns (which I maintain manually after downloading) to my title page?
And how do I update my book to do this?

Yes, it's possible.

https://manual.calibre-ebook.com/cus...ates-et-cetera

Overriding icons, templates, et cetera

calibre allows you to override the static resources, like icons, javascript and templates for the metadata jacket, catalogs, etc. with customized versions that you like. All static resources are stored in the resources sub-folder of the calibre install location. On Windows, this is usually C:\Program Files\Calibre2\resources. On macOS, /Applications/calibre.app/Contents/Resources/resources/. On linux, if you are using the binary installer from the calibre website it will be /opt/calibre/resources. These paths can change depending on where you choose to install calibre.

You should not change the files in this resources folder, as your changes will get overwritten the next time you update calibre. Instead, go to Preferences → Advanced → Miscellaneous and click Open calibre configuration directory. In this configuration directory, create a sub-folder called resources and place the files you want to override in it. Place the files in the appropriate sub folders, for example place images in resources/images, etc. calibre will automatically use your custom file in preference to the built-in one the next time it is started.

As for updating the book to insert the jacket, Polish books (keyboard shortcut P) or conversion.

The Modify ePub plugin also allows inserting jackets into ePub however, I don't know if it includes custom columns or can be customized to do so.

[JimmXinu]

New Test Version Posted

2019-05-09
- Yet another fix for tagsfromtitle on base_xenforo. Don't put whole title as a tag when no [( in title.

[Silbermieze]

Hey there. I already searched the forum as well as this thread specifically, but while I found the topics I'm looking for, I couldn't find answers (or I just didn't understand them - I don't know). So I apologize if those are already answered and please just point me in the right direction.




I'm trying to organize fanfictions in calibre with a few custom columns. Now, those columns are already set (see attachment) and most of them work with the fetched metadata, but I can't get the data for words and chapters from either site. I'm not sure if I just don't get the right "label" for it or if it doesn't work at all? I know the plugin "count pages", but I could never make sense of those numbers, so if there's a chance I can get the original data from FF.net and AO3, I'd prefer them.

Secondly, while I looked around this thread I found the possibility to exclude relationships with an "&", but I can't get that to work either. Maybe I put something in my personal.ini that clashes against each other?

Spoiler:

[defaults]

sort_ships:true
sort_ships_splits:
 [ ]*/[ ]*=>/
 [ ]*&[ ]*=>\s&\s
keep_in_order_characters:true
keep_in_order_ships:true







[www.fanfiction.net]
username:XXXXXXXX
password:XXXXXXXX
is_adult:true

extra_valid_entries:reviews,favs,follows,pairings,status,rating,words,chapters,characters
check_next_chapter:false

ships_label:Pairings
reviews_label:Reviews
favs_label:Favs
follows_label:Follows
status_label:Status
rating_label:Rated
words_label:Words
chapters_label:Chapters
characters_label:Characters

custom_columns_settings:
 status=>#status
 ships=>#pairings
 rating=>#rating
 words=>#wordcount
 chapters=>#chapters
 characters=>#characters
 favs=>#kudos
 follows=>#bookmarks
 reviews=>#comments
 genre=>#genre

 

[archiveofourown.org]
username:XXXXXXXX
password:XXXXXXXX
is_adult:true

extra_valid_entries:fandoms,freeformtags,ao3categories,comments,kudos,bookmarks
add_to_extra_valid_entries:,onlyromships,ships2chars

include_in_onlyromships:ships
exclude_metadata_post: 
 onlyromships=~[ ]*&[ ]*
 onlyromships=~[ ]*&[ ]*

include_in_ships2chars:ships.NOREPL
# NOREPL prevents any ships patterns from being applied to ships2chars.

add_to_replace_metadata:
 ships2chars=>[ ]*/[ ]*=>/
 ships2chars=>[ ]*&[ ]*=>/
 ships2chars=>([^/]+)/([^/]+)/([^/]+)/(.+)=>\1\,\2\,\3\,\4
 ships2chars=>([^/]+)/([^/]+)/(.+)=>\1\,\2\,\3
 ships2chars=>([^/]+)/(.+)=>\1\,\2

include_in_characters:characters,ships2chars

fandoms_label:Fandoms
freeformtags_label:Freeformtags
ao3categories_label:AO3 Categories
comments_label:Comments
kudos_label:Kudos
bookmarks_label:Bookmarks
warnings_label:Warnings
rating_label:Ratings
ships_label:Relationships
status_label:Status

include_in_genre:freeformtags,ao3categories
include_in_category:fandoms
extra_subject_tags:fandoms,freeformtags,ao3categories
exclude_notes:authorheadnotes,chaptersummary,chapterheadnotes,chapterfootnotes,authorfootnotes,inspiredlinks

custom_columns_settings:
 status=>#status
 ships=>#pairings
 rating=>#rating
 warnings=>#warnings
 fandoms=>#fandoms
 numwords=>#wordcount
 numchapters=>#chapters
 characters=>#characters
 ao3categories=>#categories
 kudos=>#kudos
 bookmarks=>#bookmarks
 comments=>#comments
 freeformtags=>#genre

I guess I overshoot a bit with those "extra valid entries" and "labels" (which I don't understand at all), but those were my first tries to get the metadata I want and now I don't know any more what I can and can't delete to have everything still working.



And thirdly, I have one column for genre, which works perfectly for FF.net, but not AO3 (for obvious reasons). I guess there's no further breakdown of the tags beneath characters, ships, and freeformtags? So I have to either fill in the genre manually or remove the other tags manually? Also: Can I move the tags from the original tags-section to my "genre" column so they won't appear in both sections?

[JimmXinu]

Originally Posted by Silbermieze

I'm trying to organize fanfictions in calibre with a few custom columns. Now, those columns are already set (see attachment) and most of them work with the fetched metadata, but I can't get the data for words and chapters from either site. I'm not sure if I just don't get the right "label" for it or if it doesn't work at all? I know the plugin "count pages", but I could never make sense of those numbers, so if there's a chance I can get the original data from FF.net and AO3, I'd prefer them.

For most metadata, it's usually easier to use the 'Custom Columns' tab in the GUI. That will prevent issues like you have with numChapters and numWords -- note the 'num' plus capital C and W. But I can understand wanting to have them all together.

Originally Posted by Silbermieze

Secondly, while I looked around this thread I found the possibility to exclude relationships with an "&", but I can't get that to work either. Maybe I put something in my personal.ini that clashes against each other?

You copied it from somebody who wanted both ships with and without 'friends'. Which is why it used onlyromships, but you didn't do anything with onlyromships after setting it. Example below shows excluding all ships with &amp;. That should work while not catching names with other html & entities.

Originally Posted by Silbermieze

I guess I overshoot a bit with those "extra valid entries" and "labels" (which I don't understand at all), but those were my first tries to get the metadata I want and now I don't know any more what I can and can't delete to have everything still working.

<entry>_label settings are only used inside FFF on the title page and log page--they have no effect on Calibre columns. (Well, okay, they also appear in Saved Metadata Column.)

INI entries that are in defaults.ini (and not commented out) do not need to be copied to your personal.ini. So for example, you don't need those extra_valid_entries entries. ships2chars is the only added entry for additional processing you need right now.

Our code for ffnet doesn't login, there's no need for your user/pass there. Admittedly, user/pass is needed for many other sites.

Originally Posted by Silbermieze

And thirdly, I have one column for genre, which works perfectly for FF.net, but not AO3 (for obvious reasons). I guess there's no further breakdown of the tags beneath characters, ships, and freeformtags? So I have to either fill in the genre manually or remove the other tags manually?

AO3 lets authors enter basically anything and everything. And authors do.

Normalizing AO3 tags is a subject that comes up here regularly; basically I will help with specific problems you may have, but I don't consider it my job to try and fix that design decision on AO3's part. That includes adding code to walk AO3's tag pages to get normalized tag values. I've seen stories with dozens if not hundreds of tags, I'm not adding extra network hits for them all.

Originally Posted by Silbermieze

Also: Can I move the tags from the original tags-section to my "genre" column so they won't appear in both sections?

Calibre's Tags column is set from FFF by include_subject_tags. I've shown how in the example below how to remove items from it.

You can also use the 'New Only' checkbox under Standard (and Custom) Columns tabs in FFF config to manually manage columns after initial story add. The equivalent in custom_columns_settings is a ,n after the column name.

personal.ini with changes discussed:

Spoiler:

[defaults]

# moved to defaults to apply to all sites.
is_adult:true

# changed from default values
sort_ships:true
keep_in_order_characters:true
keep_in_order_ships:true

# common std entries in defaults to apply to all sites.
# all of these could be set using FFF config GUI
# append ,n to only populate on new book add
custom_columns_settings:
 status=>#status
 ships=>#pairings
 rating=>#rating
 numWords=>#wordcount
 numChapters=>#chapters
 characters=>#characters
 category=>#categories
 genre=>#genre

[www.fanfiction.net]
# site specific values -- add_to_ mechanism adds to more general
# settings from [defaults]
add_to_custom_columns_settings:
 favs=>#kudos
 follows=>#bookmarks
 reviews=>#comments

[archiveofourown.org]
username:XXXXXXXX
password:XXXXXXXX

# exclude '&' ships -- onlyromships version was for a user who was saving both
# note: only effects AO3 here--move to [defaults] for other sites.
add_to_exclude_metadata_post:
 ships=~&

add_to_extra_valid_entries:,ships2chars

include_in_ships2chars:ships.NOREPL
# NOREPL prevents any ships patterns from being applied to ships2chars.

add_to_replace_metadata:
 ships2chars=>[ ]*/[ ]*=>/
 ships2chars=>[ ]*&[ ]*=>/
 ships2chars=>([^/]+)/([^/]+)/([^/]+)/(.+)=>\1\,\2\,\3\,\4
 ships2chars=>([^/]+)/([^/]+)/(.+)=>\1\,\2\,\3
 ships2chars=>([^/]+)/(.+)=>\1\,\2

include_in_characters:characters,ships2chars

# controls what goes in Calibre's tags column
# need to set entire value to remove entries category & genre
include_subject_tags: extratags, characters, ships, status

exclude_notes:authorheadnotes,chaptersummary,chapterheadnotes,chapterfootnotes,authorfootnotes,inspiredlinks

# site specific values
add_to_custom_columns_settings:
 kudos=>#kudos
 bookmarks=>#bookmarks
 comments=>#comments

[Silbermieze]

Thank you so much for your help. There's still a lot of work left, especially with those AO3 tags, but I'm grateful for what I can do with FFF.

Originally Posted by JimmXinu

That will prevent issues like you have with numChapters and numWords -- note the 'num' plus capital C and W.

numWords and numChapters were actually my first tests, but all in lower case. Explains a lot.

Originally Posted by JimmXinu

Normalizing AO3 tags is a subject that comes up here regularly; basically I will help with specific problems you may have, but I don't consider it my job to try and fix that design decision on AO3's part.

I didn't mean to imply you should implement a more detailed breakdown of those tags into your plugin. Just wanted to know if AO3 has a more detailed breakdown on their site and I just didn't find it yet. Sorry if my phrasing was unclear.

[JimmXinu]

Originally Posted by Silbermieze

I didn't mean to imply you should implement a more detailed breakdown of those tags into your plugin. Just wanted to know if AO3 has a more detailed breakdown on their site and I just didn't find it yet. Sorry if my phrasing was unclear.

No problem. Sorry I came across gruffly about it; it's a subject that comes up regularly enough I have a reflexive response.

AO3 does in fact have tags that are declared synonyms of other tags, but they still have waaaay too many tags, both on stories and in general. Looking at the Latest page, a story with only 10 tags is below average.

And here's a random tag example: the tag "Minor Character Death" has 302 synonyms.

(where did this soapbox come from?)

[Tanjamuse]

Originally Posted by JimmXinu

(where did this soapbox come from?)

I'm pretty sure I handed you some of the wood for it with all my questions. ...

[readerboy7]

I have a feature request regarding A03 character tags (sorry).

When downloading the tags, for the title page, would it be possible to keep the tags in the same order as they were on A03 instead of sorting them in alphabetical order? I understand that that's probably not possible on Calibre, but would it be possible in the title page?

Because users put stuff in tags that should really be in the summary, which makes the order of tags sometimes matter.

E.g.

For https://archiveofourown.org/works/17...pters/42011579

desired output:*

Genre: Reincarnation, but more literally and also more than canon, Crack, ...

actual output

Genre: Alternate Universe - Canon Divergence, Angst, BAMF Dai-nana-han | Team 7 (Naruto), ...

*Okay, actual desired output would be for A03 to have a decent tag system, but that isn't your job, and I'm not asking you to fix it.

[JimmXinu]

Originally Posted by Tanjamuse

I'm pretty sure I handed you some of the wood for it with all my questions. ...

No worries.

Originally Posted by readerboy7

When downloading the tags, for the title page, would it be possible to keep the tags in the same order as they were on A03 instead of sorting them in alphabetical order? I understand that that's probably not possible on Calibre, but would it be possible in the title page?

The example in defaults.ini isn't great, but the feature does exist:

## From defaults.ini:
## keep_in_order_<entry> options: FanFicFare sorts list entries by default
## (except for author/authorUrl/authorId). But if you want to use an
## extra entry derived from author, it ends up sorted. For example, if
## you added calibre_author: keep_in_order_calibre_author:true
#keep_in_order_author:true

## add to your personal.ini in section [archiveofourown.org]

# collected from AO3:
keep_in_order_freeformtags:true
keep_in_order_ao3categories:true
keep_in_order_fandoms:true

# These will still be sorted if you don't keep_in_order_ them too
# genre is freeformtags + ao3categories
keep_in_order_genre:true

# category is fandoms
keep_in_order_category:true

[JimmXinu]

New Test Version Posted

2019-05-12
- Fixes for adapter_storiesonlinenet site changes--login and use datetime by default because one was missing date-only.
- adapter_finestoriescom shares code with adapter_storiesonlinenet, also do datetimes.
- Comment out some debugs in XF/XF2

[ILB]

Originally Posted by JimmXinu

Calibre's Polish feature has an option for add/update (and one for remove) 'metadata jacket'. The jacket is placed directly after the cover (if present). It does not effect the FFF title page, nor does it appear in the TOC like FFF title page does in multi-chapter books.

You can customize it by copying the default jacket template directory to your personal resources dir and then editing the files. The manual page explains it, but it may be a little out of date as for me the source dir is C:\Program Files\Calibre2\app\resources\jacket . Custom columns can be added by using _ instead of #. For #genre:

        <div>
          <p class="cbj_label">{_genre_label}:</p>
          <p class="cbj_content">{_genre}</p>
        </div>

FFF on update will discard the jacket file. But FFF already has an option to invoke the Polish option to smarten punctuation and I would be willing to add a similar option to automatically call Polish update jacket.

UPDATE: Calling Polish to update jacket might be more difficult than that--there are some 'interesting' interactions between conversion and jacket I would need to explore more.

(BTW, note that if you use Polish to add/update cover page, it will name the cover page file titlepage.xhtml. FFF's title page (title_page.xhtml) is unchanged.)

Thanks JimmXinu and ilovejedd, I tried polish but it gives me a whole extra page that I don´t need – nor want.
It also copies my epub in “Original-EPUB” and adds the new title page in “EPUB” – which makes two EPUBs for which I need memory space when I want only one epub.

It´s only one maybe two of my custom columns that I want to add.
I liked the existing option to add columns to the “normal” title page. And all columns that are filled by settings in personal.ini already work when downloading.
It is just the hassle with columns that are manually filled after downloading.