Recognizing KFX virtues

[PoP]

Just to report that in this instance KFX is the format which rendered best my text:

Inspired by a recent link in the workshop,

Originally Posted by Doitsu

...
Here's a link to a German website that contains hi res scans of the original German book.
...

I reproduced the ebook with a Google french translation of the german text.

To give it a Germanized look I decided to code the text using the "Fraktur Bold" letters in the "Mathematical Alphanumeric Symbols" of the Unicode block (U+1D400..U+1D7FF) embedding a publisher font since no native font of the Kindle support this block. Not using Unicodes from the "Basic Multilingual Plane" (BMP) proved a challenge...

Expectedly, this MOBI badly tricked the K3:


On the PW3 the MOBI introduced unwanted blanks after accenting Fraktur letters with diacritical combining marks:


But converting the MOBI to KFX perfectly fixed the rendering:


Find my sources here:

[lumpynose]

Perhaps using the '"Mathematical Alphanumeric Symbols" of the Unicode block (U+1D400..U+1D7FF)' is not a best practice for regular running text. What about using a Fraktur font? E.g., https://fonts.google.com/specimen/UnifrakturCook

[PoP]

Originally Posted by lumpynose

Perhaps using the '"Mathematical Alphanumeric Symbols" of the Unicode block (U+1D400..U+1D7FF)' is not a best practice for regular running text. What about using a Fraktur font? E.g., https://fonts.google.com/specimen/UnifrakturCook

You are correct, but where is the fun

My point was to use a code block that would not change the "Germanized" look even if the user would change in his preferences the viewing font. At the beginning, I was naively expecting the Kindle fonts to support that code block natively. Of course my idea failed when I needed to embed my own font.

Coding the book with characters out of the BMP range, and converting to html entities was a fun struggle anyway.

[Tex2002ans]

Originally Posted by lumpynose

Perhaps using the '"Mathematical Alphanumeric Symbols" of the Unicode block (U+1D400..U+1D7FF)' is not a best practice for regular running text. What about using a Fraktur font?

Exactly.

The Fraktur Unicode characters are meant only for Maths (Formulas, Equations, Variables, [...]).

https://en.wikipedia.org/wiki/Mathem...umeric_Symbols

Especially see the 2nd paragraph:

Mathematical Alphanumeric Symbols are Unicode blocks of Latin and Greek letters and decimal digits that enable mathematicians to denote different notions with different letter styles. The letters in various fonts often have specific, fixed meanings in particular areas of mathematics. By providing uniformity over numerous mathematical articles and books, these conventions help to read mathematical formulae.

[...]

Unicode expressly recommends that these characters not be used in general text as a substitute for presentational markup; the letters are specifically designed to be semantically different from each other. [...]

Also see the "Fraktur" article on Wikipedia:

Individual Fraktur letters are sometimes used in mathematics, which often denotes associated or parallel concepts by the same letter in different fonts. For example, a Lie group is often denoted by G, while its associated Lie algebra is 𝖌. [...] In model theory, 𝕬 is used to denote an arbitrary model, with A as its universe. Fraktur is also used in other ways at the discretion of the author.

Originally Posted by PoP

My point was to use a code block that would not change the "Germanized" look even if the user would change in his preferences the viewing font.

This is extremely poor practice and would completely ruin things like Text-to-Speech. Properly encoded characters are very important.

Side Note: A similar issue cropped up with the Hebrew Aleph character in this topic:

https://www.mobileread.com/forums/sh....php?p=3574570

That user accidentally used the Hebrew Alef א (U+05D0) instead of the Alef Symbol ℵ (U+2135).

It broke the layout of the book because he used the wrong character.

Originally Posted by PoP

At the beginning, I was naively expecting the Kindle fonts to support that code block natively. Of course my idea failed when I needed to embed my own font.

Most of that mathematical block was added in Unicode 3.1, but support is really rare... mostly focused on fonts designed specifically for Maths... like the STIX fonts:

https://www.stixfonts.org/

You can also use a tool like BabelMap in order to find out which fonts on your computer include those characters:

http://www.babelstone.co.uk/Software/BabelMap.html

Complete Side Note: This reminds me of the "Look-Alike Domains" that unscrupulous spammers/malware-producers use to trick people into going to their domains or clicking on links or trying to get around spam blockers:

https://krebsonsecurity.com/2018/03/...ual-confusion/

Another thing that many scammer "ghost writers" on places like Fiverr do is run text through programs which try to substitute original characters with their "visual equivalents" to try to get around plagiarism checkers.

Here's some examples:

https://forum.fiverr.com/t/article-w...m-mafia/160631
https://forum.fiverr.com/t/scam-plag...lers/146821/15

An easy way to spot this is typically your spellchecker going insane with red squigglies everywhere.

Long story short... don't do this. Just include a Fraktur font for visual distinction.

[PoP]

@lumpynose, @Tex2002ans,

Thanks for the your comments and suggestions on best practices; I would never advocate to do otherwise.

I was just playing with the Unicode as a fun experiment.

Nonetheless my point on KFX better rendering remains.

[lumpynose]

(I don't know anything about creating epubs.) Out of curiosity, is there some html that you use to add the accents to the letters or do you use predefined letters that have accents? In previous life I used the TeX typesetting software and with it you could add accents and whatnot to any letter with a special sequence after the letter, I think. This was decades before Unicode.

[jhowell]

PoP, are you a Mac user?

I ask because I have been unable to create books in KFX format that use unicode characters beyond plane 0 on a PC. If you are using a PC I would like to know more about how you made that work.

[lumpynose]

Originally Posted by jhowell

PoP, are you a Mac user?

I ask because I have been unable to create books in KFX format that use unicode characters beyond plane 0 on a PC. If you are using a PC I would like to know more about how you made that work.

But you can with azw3 or mobi?

If you're on Windows there's a thing called Character Map. As an experiment you could try copying a character from it and pasting it into whatever you're using, just to verify that it's not a keyboarding issue.

[PoP]

Originally Posted by jhowell

PoP, are you a Mac user?

I ask because I have been unable to create books in KFX format that use unicode characters beyond plane 0 on a PC. If you are using a PC I would like to know more about how you made that work.

I am using a PC with Windows 7.

The zip attached to the OP contains the sources that I used as input to Kindlegen. The first pages of the book detail (but in french) the method that I used:

Spoiler:

Basically, you can't code non-BMP characters directly. I used a text editor to create "Struwwelpeter fraktur.html" and then used BabelPad's Tools To convert Unicodes to NCR, producing "Struwwelpeter.html". This feeds happily Kindlegen as well as any browser.

After that I used Caliber and *your* KFX output plugin to generate the KFX book format.

[PoP]

Originally Posted by lumpynose

(I don't know anything about creating epubs.) Out of curiosity, is there some html that you use to add the accents to the letters or do you use predefined letters that have accents? In previous life I used the TeX typesetting software and with it you could add accents and whatnot to any letter with a special sequence after the letter, I think. This was decades before Unicode.

Nothing special really.

Usually, I just use dead keys on an International Keyboard to input pre-composed characters. e.g. ' followed by e gives é

In this case (fraktur) I used a combination of YayText and WizKey¹ after the fraktur character to generate the diacritical combining marks, to accentuate them.


¹ With a crazy big key definition file that I developed along the years (btw no promotion here - I am not affiliated in any way with WizKey)

[jhowell]

Originally Posted by PoP

Basically, you can't code non-BMP characters directly. I used a text editor to create "Struwwelpeter fraktur.html" and then used BabelPad's Tools To convert Unicodes to NCR, producing "Struwwelpeter.html". This feeds happily Kindlegen as well as any browser.

After that I used Caliber and *your* KFX output plugin to generate the KFX book format.

Thanks. I am able to get it to work by following the same steps that you used.

Oddly, if I skip using kindlegen or the calibre GUI it fails for me. I need to do some research into why only some conversions methods work properly.

[jhowell]

Originally Posted by jhowell

I need to do some research into why only some conversions methods work properly.

After some research I was able to determine the reason why unicode characters outside of plane 0, such as Fraktur characters, work in the Struwwelpeter book but did not in my prior testing.

There is a strange bug in the KFX converter within the Kindle Previewer that causes these characters to be rejected unless they are preceded in the book by at least one character from a right-to-left script. In this case the backwards "Copyleft" used near the beginning of the book includes a Hebrew character (nun) that prevents the bug from manifesting.

[PoP]

Originally Posted by jhowell

[...] includes a Hebrew character (nun) that prevents the bug from manifesting.

Oh, I had beginner's luck! Thanks for your analysis. Kudo, I am impressed that you could find this with no source.

Note to myself: always open source with a ʇɟǝןʎdoƆ (ↄ) intro. U+05DF

[Tex2002ans]

Originally Posted by PoP

Note to myself: always open source with a ʇɟǝןʎdoƆ (ↄ) intro.

The actual copyleft character 🄯 (U+1F12F) was added in Unicode 11.0 (June 2018).

[PoP]

Originally Posted by Tex2002ans

The actual copyleft character 🄯 (U+1F12F) was added in Unicode 11.0 (June 2018).

Yeah, I know, but you do need to embed a font to make the glyph readable. I'll stick to (ↄ) for now. It is the U+05DF that is essential here.