[GUI Plugin] FanFicFare

[ownedbycats]

A fellow fanfic writer I'm friends was kind of annoyed at it because she had 80-some tags on one fic. Ended up merging a few.

[JimmXinu]

Originally Posted by ownedbycats

They actually did! Limit of 75 tags now, after that incident with the story using literally thousands of tags and outright crashing browsers. But older works with more than that will keep the tags (until the author tries to update, then it'll make them delete some).

I did not know that. A quick google finds the AO3 announcements.

Why 75 tags?

We looked at all the works on the Archive and determined that:

the average number of tags per work is 17,
the most common number of tags per work is 11, and
less than 0.5% of works have more than 75 tags.

A limit of 75 total fandom, character, relationship, and additional tags per work will provide the vast majority of creators with enough space to describe the content of their works while also keeping work blurbs to a manageable size.

[ownedbycats]

Originally Posted by Amalthia

I think I figured it out.

Here is what worked:



I'm not sure what =~ means but I know != excluded all genre's except the ones that started with Additional.

I'm going through the calibre regex reference guide to see if there is a list of these selectors.

Amalthia

Also I note that you excluded the ^ anchor at the beginning. Just a small warning: This means it'll also delete things where "additional" is in the middle of the tag.

[mabelrxu]

I'm following the PluginWithReadingList guide for getting fanfics from ao3 with a small tweak (I have it say [new] instead of 000) but I'd like some help setting up a variation ...

If the fanfic is read, I'd like to remove it from my device. That way, no plugboard with [new] needed, since the books on Kindle have that new vs read label on them anyways. Also that'll keep my kindle from cluttering up with too many books.

On a side note, any way to have the reading status of the book on kindle checked to automatically mark a book as read (mark as read the fanficfare way, aka mark read and add to send to device list)?

On a side side note, any ways to add fanfics to be tracked to fanficfare without sitting down at my computer to copy and paste in URLs?

[ownedbycats]

Originally Posted by mabelrxu

On a side side note, any ways to add fanfics to be tracked to fanficfare without sitting down at my computer to copy and paste in URLs?

Do the websites you get fanfics from have email notifications? There's option to fetch via imap. Personally, I drag and drop from Mozilla Thunderbird onto the download-from-url window.

[JimmXinu]

Originally Posted by mabelrxu

I'm following the PluginWithReadingList guide for getting fanfics from ao3 with a small tweak (I have it say [new] instead of 000) but I'd like some help setting up a variation ...

If the fanfic is read, I'd like to remove it from my device. That way, no plugboard with [new] needed, since the books on Kindle have that new vs read label on them anyways. Also that'll keep my kindle from cluttering up with too many books.

Hmmm. It's not really designed to do that.

I believe you could do it by configuring the 'To Read' list to "Add New to device" when sync'ing, and the 'Send to Device" list to "Remove from device" when sync'ing, with "Add New/Update to 'Send to Device' List" UNchecked and "Add books back to 'Send to Device' List when Read" checked.

But I haven't tested that.

Originally Posted by mabelrxu

On a side note, any way to have the reading status of the book on kindle checked to automatically mark a book as read (mark as read the fanficfare way, aka mark read and add to send to device list)?

I don't believe that's possible. FFF certainly isn't coded to do it.

Originally Posted by mabelrxu

On a side side note, any ways to add fanfics to be tracked to fanficfare without sitting down at my computer to copy and paste in URLs?

Since you seem to be focused on wanting new/updated fics on your device, I'd suggest the FFF email integration to semi-automatically add stories to Calibre (using FFF) as you get update notifications for them.

[Amalthia]

Originally Posted by ownedbycats

Also I note that you excluded the ^ anchor at the beginning. Just a small warning: This means it'll also delete things where "additional" is in the middle of the tag.

I'm debating if there are any useful genre tags that have "additional" in the middle of the phrase?

I've been normalizing genre tags as I run across them. There is a lot of variety I had 300 different versions of "Fix It" genre tags. The hard part is determining what metadata is worth keeping. I think for now I'm going to slowly add stuff as needed. I started with over 60k genre tags and I'm now at 54k tags. If I can get the tags under 50k by normalizing the data I'll be happy.

I think it's better to use "Fix-It," "Golden Core," "Sunshot Compaign," as separate pieces of metadata vs mushing it all together.

I'm now curious to know how other people use calibre to normalize their metadata, only I'm not sure this thread is the place to find out?

Waves,
Amaltha

[JimmXinu]

Originally Posted by Amalthia

I'm now curious to know how other people use calibre to normalize their metadata, only I'm not sure this thread is the place to find out?

A short discussion here is fine. But it if goes on for long, I'd suggest starting a thread under Library Management.

IMO, normalizing AO3 tags is a fundamentally unending task. All I do is remove the standardized 'not' tags:

exclude_metadata_pre:
 freeformtags,genre==Other Additional Tags to Be Added
 warnings==Creator Chose Not To Use Archive Warnings
 warnings==No Archive Warnings Apply

Although I'm not sure where 'Other Additional Tags to Be Added' came from.

[watani]

I'm guessing RoyalRoad has changed its website as of today? Every story I try to update returns the following:

'NoneType' object has no attribute 'text'

I'm using the latest test version of FFF.

[aleyx]

Originally Posted by watani

I'm guessing RoyalRoad has changed its website as of today? Every story I try to update returns the following:

'NoneType' object has no attribute 'text'

I'm using the latest test version of FFF.

As it happens, I'm looking into this myself.

Seems that adapter_royalroadcom.py looks for the chapter title with the selector '.fic-header h1[property=name]' (line 165), but the h1 now only has class and style, so the selector fails (if I understand correctly) (kind of weird, I dont remember selectors with that syntax, but oh well, it worked til now).

PHP Code:

title = soup.select_one('.fic-header h1[property=name]').text 


The title is therefore "None", and None has no attribute 'text', thus the error message.

I was afraid at first, because the first few error messages I got were HTTP 403, "Client forbidden". Was afraid that RoyalRoad now excluded bots completely.

N.

EDIT: For reference, the traceback I got

Spoiler:

	Traceback (most recent call last):
	  File "calibre_plugins.fanficfare_plugin.jobs", line 250, in do_download_for_worker
	    story = adapter.getStoryMetadataOnly()
	  File "/home/aleyx/.var/app/com.calibre_ebook.calibre/config/calibre/plugins/FanFicFare.zip/fanficfare/adapters/base_adapter.py", line 327, in getStoryMetadataOnly
	    self.doExtractChapterUrlsAndMetadata(get_cover=get_cover)
	  File "/home/aleyx/.var/app/com.calibre_ebook.calibre/config/calibre/plugins/FanFicFare.zip/fanficfare/adapters/base_adapter.py", line 431, in doExtractChapterUrlsAndMetadata
	    return self.extractChapterUrlsAndMetadata()
	  File "/home/aleyx/.var/app/com.calibre_ebook.calibre/config/calibre/plugins/FanFicFare.zip/fanficfare/adapters/adapter_royalroadcom.py", line 165, in extractChapterUrlsAndMetadata
	    title = soup.select_one('.fic-header h1[property=name]').text
	AttributeError: 'NoneType' object has no attribute 'text'
	
	FFF: INFO: 2023-02-24 20:46:03,346: calibre_plugins.fanficfare_plugin.jobs(142): 
	Download Results:
	Error https://www.royalroad.com/fiction/41463 'NoneType' object has no attribute 'text'
	
	FFF: INFO: 2023-02-24 20:46:03,346: calibre_plugins.fanficfare_plugin.jobs(166): 
	Error
	https://www.royalroad.com/fiction/41463

EDIT:

So I've made two modifications to adapter_royalroad.py:

Spoiler:

Line 165: Changed to

        title = soup.select_one('.fic-header .fic-title h1').text

Line 194: Changed to

        description = soup.select_one('div.description .hidden-content')

I've successfully updated manually and from email. That was kind of field surgery though and I know nothing about BeautifulSoup, so the adapter's author should maybe check that those changes are in fact harmless.

[dailyalice]

Originally Posted by Amalthia

I'm now curious to know how other people use calibre to normalize their metadata, only I'm not sure this thread is the place to find out?

Giving up entirely on AO3's freeform tags is a reasonable thing to do (you don't even have to download them).

I stopped working on my guide to FanFicFare and calibre for a few months and I suspect some of that was because I started a section on normalizing metadata and got a bit overwhelmed. There's a ton you can do! But is it worth it? Key is figuring out which data you find most useful and would find useful in your library.

As far as AO3's tags go, I have two columns in my fic library that are filled using freeformtags.

The first column includes all of the messy AO3 tags: I put them in a separate column and don't touch them. So they're there if I want to search them or use them, but I can also easily ignore them.

I have this under add_to_custom_columns_settings in the AO3 section of my personal.ini:

 freeformtags=>#freeformtags

And I have a custom column in calibre of the tag type with a lookup name of "freeformtags" and column heading of "Tags (AO3)"

But I also use the freeformtags somewhere else. In the [defaults] section of my personal.ini, I have:

custom_columns_settings:
 tags=>#tags

include_in_tags:freeformtags

tags_label:Tags (Edited)

And I've created yet another custom tags column in calibre for that data (tags type, lookup name of tags, column heading of "Tags (Edited)" though I'm not wild about that name and may change it.

And then I do some tag normalization under replace_metadata using the metakey of "tags" to clean up some of those freeformtags.

Thrilling stuff like:

 tags=>^[Ff]luff [Aa]nd [Aa]ngst$=>Fluff\,Angst
 tags=>^[Aa]ngst [Aa]nd [Hh]umo(u)?r$=>Humor\,Angst
 tags=>^[Aa]ngst [Aa]nd [Ff]luff$=>Fluff\,Angst
 tags=>^[Ff]langst$=>Fluff\,Angst

I also do some stuff with tag hierarchies so tags that are specific to certain fandoms wind up being in tag hierarchies. MCU.Post-Endgame, MCU.Shrunkyclunks, and stuff like that. A person can easily go overboard.

But the real key to the whole enterprise is that then I use include_metadata_post to indicate the only tags I want to see in that Tags (Edited) column. They can be some that I normalized a bit using replace_metadata and some of the more useful ones from AO3 that I keep as is.

include_metadata_post:
 tags==Action/Adventure
 tags==Angst
 tags==Crack
 tags==Drama
 tags==Established Relationship
 tags==Fluff
 tags==Humor

You can choose to keep whichever tags from the AO3 freeformtags that you find most useful. If you already have a library, you can use the tag browser to see which tags are used most often; there are also lists online of the most popular tags on AO3. They can be a good starting point, though most folks know which tags they find most helpful.

In one of my bigger calibre fic libraries, I have over 30,000 tags in my Tags (AO3) column (GOOD GRIEF) and about 400 in my Tags (Edited) column.

I've kept these all separate from calibre's main tags column/field which I use for other things.

(Step by step instructions, with screenshots & code you can copy, will be in my guide if I ever get it done and figure out how/where to post it.) (The Google doc was well over 500 pages when last I checked and then there are separate docs on other topics like normalizing metadata.)

[ownedbycats]

EDIT: Discussion moved here as requested by Jim

[JimmXinu]

New Test Version Posted

2023-02-24
- adapter_royalroadcom: Fixes for site changes. #923

Needed changes for title, description and genre.

[ravendm5]

Not sure if anyone else is having this problem, but spacebattles has stopped working for me. It asks for a login, but doesn't download anything even after it is entered.

EDIT: Please disregard. My VPN was left on and that seems to interfere with spacebattles.

[JimmXinu]

Originally Posted by ravendm5

Not sure if anyone else is having this problem, but spacebattles has stopped working for me. It asks for a login, but doesn't download anything even after it is entered.

It's working for me. What's your story URL that's failing?

It's failing for all stories, have you been downloading a lot? Or trying to do more than one job at a time? You might be getting throttled by the site.

UPDATE: Or VPNs can cause problems.