[GUI Plugin] FanFicFare

[JimmXinu]

Originally Posted by Tanjamuse

Yes that's it. I would love it to work with Custom columns as well, I don't know if that's possible but the tags are a great way to start. It would save me from trying to weed out all those run-on sentences.

Why would that automatically save you "trying to weed out all those run-on sentences"? At first glance, it's not as capable as what FFF provides (albeit more GUI/user-friendly).

Also, Calibre has an apparently identical, but separate set of rules you can configure for Adding books.

Which would actually be appropriate to apply? Or one for new books and the other for updates? Or using the same Calibre dialog & code to make a FFF specific set of rules?

And are any other users interested in having this?

EDIT: Scary discovery--the metadata download / add book tag filter rules are not library specific...

[Kern]

Originally Posted by JimmXinu

And pixiv is in Japanese. It's difficult to code for a language if you don't understand it.

Fair enough. Pixiv does have an English option for the site, with some English fics however.

Originally Posted by JimmXinu

The best way to get a site supported is to write the code for it yourself. It's an open source project, after all.

Is there documentation for that? I have a vague idea what to do looking at the source.

[JimmXinu]

Originally Posted by Kern

Is there documentation for that? I have a vague idea what to do looking at the source.

There isn't a document as such. I should probably make one some day.

The key steps of creating an adapter are:

• Create the fanficfare/adapters/adapter_<site>.py file, probably by copying an existing one
• Add an import to fanficfare/adapters/__init__.py for the new file
• Change the class name and siteabbrev value
• Change the code to normalize story URLs
• Change site example urls
• Change the regex code to recognize story URLs
• Change the code to fetch, parse and save metadata (extractChapterUrlsAndMetadata)
• Change the code to login to the site, if needed
• Change the code to fetch and parse each chapter text (getChapterText)
• Add entries for the site to plugin-defaults.ini & defaults.ini if the adapter has settings.

Differences in sites makes it hard to choose which adapter to start from, but this adapter has some comments about making a new one.

The code is in GitHub. You can fork your own copy of the repos, work there and then create a Pull Request I can merge in.

[ilovejedd]

Originally Posted by JimmXinu

EDIT: Scary discovery--the metadata download / add book tag filter rules are not library specific...

I believe one can save/load rulesets. I'm not sure if the rulesets are universal or per library so perhaps one needs to remember to set the correct ruleset when switching libraries when adding books.

Because the author_sort tweak is universal instead of per library, I actually run calibre portable and have two completely separate instances for regular books and fanfiction.

I don't believe it will work for custom columns, though. I'm currently testing a refactor of my entire AO3 library (well, a full copy of metadata.db anyway so I can capture all tags in use - both good and bad/freeform).

Current plan is:
#ships_freeform -> tags -> Tag Mapper (ships ruleset) -> #ships_clean -> nuke tags (so it can be used with other tags)

Also planning on trying CALM plugin which can work directly on custom columns but reading the instructions, it looks like setting that up will take a bit longer so I'm saving that for the weekend.

[kovidgoyal]

Since tags and authors are common to all libraries, the rulesets are also common to all libraries. If you want to use them for custom colunm, I would suggest re-using the infrastructure, for creating/applying the rules, but use your own code to store the rulesets and load them as needed.

[JimmXinu]

I did look at this, both applying one of the existing rule sets and re-using the infrastructure (aka tag_mapper) to create additional rule sets. These are my observations:

• Applying the existing add book or download metadata rule sets is problematic. Which? When? FFF configures everything by library, those rules do not.
• Creating new rule sets using tag_mapper/author_mapper are possible per library. But for just tags/authors? For custom columns? For each custom column?
• Pro: The tag_mapper a nice GUI solution for filtering and modification of tags.
• Con: It's not as flexible and capable as FFF's ini text solution. For example, conditionals have no equivalent.
• Pro: Aside from GUI-ness, the biggest advantage I see to the tag_mapper is that it uses the Tag Editor, which knows what tags are already in the library, making it easy to limit incoming tags to those already in the library.
• Con: When I created a rule containing all my tags (1867), I wasn't able to edit that rule again reliably. Double clicking it only occasionally worked.
• Con: Redundancy of features: I couldn't remove replace_metadata from FFF even if I wanted to because CLI and web service still need it. Troubleshooting complex setups with both is daunting.

And finally, only Tanjamuse and (maybe?) ilovejedd showed any interest.

So I'm shelving this proposed change for the time being. A more general tool, like CALM may be useful.

[JimmXinu]

New Test Version Posted

2018-10-03
- Handle stories without cover correctly adapter_wwwlushstoriescom. Closes #344

[fanfictioneer]

Feature request:
De-Zalgo-ifying text

Reason:
My Kindle does not handle large chunks of Zalgo-ified text gracefully. I noticed this in the fanfic Abaddon Borne - basically, my Kindle just gives up when it reaches the Zalgo-ified chunk of text. It handles smaller pieces without a problem, but I think it is when it reaches several rows that the problem really occurs.

A super-graceful way of doing it would be to insert "(Zalgo-text starts)" and "(Zalgo-text ends)" at every Zalgo-chunk which is longer than X (characters/words), but that is definitely not what I need - just an option to remove the Zalgo-ified-text.

Python code that I use in my modified "Modify Epub" plugin to remove it (originally from Stack Overflow):

import unicodedata

ZALGO_CHAR_CATEGORIES = ['Mn', 'Me']

stripped = ''.join([c for c in unicodedata.normalize('NFD', text) if unicodedata.category(c) not in ZALGO_CHAR_CATEGORIES])

[JimmXinu]

Originally Posted by fanfictioneer

Feature request:
De-Zalgo-ifying text
...

That's a very good idea, actually. I've bumped into that myself, but I never looked into it in any depth.

However, rather than stripping Zalgo entirely, I wonder if it could be reduced to a minimum amount that would work in readers while still getting the point across?

Or a limit number for it that could be set by the user to 0 to remove it entirely.

I'll look into it.

[fanfictioneer]

Thanks!

That would certainly work, too - the Kindle have no issues with minor Zalgo-fication, but I suspect it runs into issues when it runs into several lines - then it starts to slow down and becomes unresponsive.

Something like this should work for your needs (warning: I am a Python n00b, and rusty on top of that):

import unicodedata

ZALGO_CHAR_CATEGORIES = ['Mn', 'Me']
maxZalgo = 0
currentZalgo = 0

def deZalgo(c):
	isZalgo = unicodedata.category(c) in ZALGO_CHAR_CATEGORIES
	if isZalgo:
		currentZalgo += 1
		if currentZalgo > maxZalgo:
			currentZalgo = 0
			return False
	return True

stripped = ''.join([c for c in unicodedata.normalize('NFD', text) if deZalgo(c)])

[JimmXinu]

New Test Version Posted

2018-10-10
- Add max_zalgo feature.

## Some authors use 'Zalgo' text--arbitrary and often excessive
## added/combined unicode markings--to indicate 'noise' of some kind.
## While a critical part of some languages, when over used it can also
## cause problems for some ebook readers, such as Kindle.
## https://stackoverflow.com/questions/6579844/how-does-zalgo-text-work
## This setting will reduce the number 'Zalgo' characters.  Set to 0
## to remove all of them, -1 or leave undefined for no limit.
## Note: This will also remove other 'marked' unicode characters. If
## set to 0, café will become cafe, déjà -> deja, etc.  1 will keep
## single marks and is the recommended setting if you use it.
#max_zalgo:1

[readerboy7]

Whenever I try to download a fic from FFnet, I get the following error:
NoneType' object has no attribute '__getitem__'

This seems to apply to any FFnet link, but I'm giving an example link now as standard practice: https://www.fanfiction.net/s/8873327/1/

The error does not seem to appear with other websites, such as AO3 and SB.

I am running FFF version 3.0.0, calibre version 3.28

[ilovejedd]

Originally Posted by readerboy7

Whenever I try to download a fic from FFnet, I get the following error:
NoneType' object has no attribute '__getitem__'

This seems to apply to any FFnet link, but I'm giving an example link now as standard practice: https://www.fanfiction.net/s/8873327/1/

The error does not seem to appear with other websites, such as AO3 and SB.

I am running FFF version 3.0.0, calibre version 3.28

I'm getting random Error Type 1 messages with Firefox so perhaps just some intermittent website issues?

[JimmXinu]

Originally Posted by readerboy7

Whenever I try to download a fic from FFnet, I get the following error:
NoneType' object has no attribute '__getitem__'

This seems to apply to any FFnet link, but I'm giving an example link now as standard practice: https://www.fanfiction.net/s/8873327/1/

The error does not seem to appear with other websites, such as AO3 and SB.

I am running FFF version 3.0.0, calibre version 3.28

I'm not seeing errors with ffnet. I was able to download that story.

If you're still seeing errors, you can post a debug log (run Calibre in debug, under Preferences drop down) and I'll look at it.

But otherwise, I assume you're seeing a temporary site issue like ilovejedd said.

[JimmXinu]

New Test Version Posted

2018-10-11
- Fix adapter_wuxiaworldco date format for 24 hour clock. Closes #345