Font hack to replace code2000.ttf?

[Doitsu]

Since my old K3 firmware didn't support embedded fonts that replace Unicode ranges covered by code2000.ttf, I decided to get a PW2.

Unfortunately, Amazon still hasn't fixed this issue. While my K3 wouldn't display Arabic glyphs from embedded fonts at all, my PW2 does display Arabic glyphs right-to-left, but doesn't do glyph shaping at all.

Basically, it displays Arabic text like the text in the second line of this Wikipedia image (i.e. right-to-left, but with isolated letter forms instead of shaped letter forms):



If I disable the Publisher Font option (by selecting Caecilia as the default font) Arabic text is displayed using the glyphs from the code2000.ttf fallback font, which are really hard to read.

It'd be great if someone with USB network access to their Kindle PW could do the following simple test:

1. Rename code2000.ttf to code2000.bak in the Kindle fonts folder.
2. Copy Arial Unicode MS (ARIALUNI.TTF) from your Windows machine to the Kindle fonts folder and rename it to code2000.ttf or modify whatever file in /etc/fonts/conf.d needs to be modified so that the replacement font is used instead of code2000.ttf for all glyphs not covered by the default fonts. (If you have a Linux machine, just use the old default Arial font (arial.TTF) from the Core fonts for the Web package, which come pre-installed with most distros.)
3. Copy my test file to the \documents folder on your Kindle, open it and disable the Publisher Fonts option or select Caecilia as the default.

If the Kindle then displays the Arabic text correctly with the glyphs from the Arial font, I'd really appreciate it, if someone could package a noob friendly font hack .bin file that'll automatically replace code2000.ttf with another user-defined fallback font.

I'm sure that many Kindle users who speak (or learn) languages that use the Arabic alphabet (e.g. Arabic, Persian, Sorani Kurdish, Urdu etc.) would be very grateful for this.

[NiLuJe]

IIRC, the rendering engine still doesn't handle complex script shaping, no matter the font used, even in KF8 files (as you saw with the embedded font test).

And if the goal was just to live with that but with another font than code2000, it's slightly more complex than on the K3, just replacing code2000 won't do a thing (besides potentially breaking stuff in weird & interesting ways ;p).

I'll look into how exactly it's handled, see if I can come up with something like the font overrides for sans/serif/mono in the Fonts Hack.

[Doitsu]

Originally Posted by NiLuJe

IIRC, the rendering engine still doesn't handle complex script shaping, no matter the font used, even in KF8 files (as you saw with the embedded font test).

AFAIK, there are two different rending engines: one for mobi7 files and one for mobi8 (KF8/azw3) files.

Before the final K3 firmware update, my K3 couldn't display RTL text at all. After the 3.4 firmware it would properly display Arabic text using the glyphs from the code2000.ttf font.

My K3 displays Arabic text exactly as my PW2 (see PW2_RTL_PubFontOff.png in my first post). I.e. the script shaping definitely works as designed, at least with the glyphs from code2000.ttf.

That's why I was wondering if simply replacing code2000.ttf would do the trick, unless the Kindle firmware does some kind of font checksum test.

IIRC, /etc/fonts/conf.d contains several configuration files that appear to define the fonts to be used for the different Unicode ranges.

IIRC, the configuration file for the fallback font listed Arial Unicode MS before Code2000. If the order in which the fonts are listed is the order in which they're processed, maybe simply copying Arial Unicode MS (or an Open Source font with better Arabic glyphs) to the internal Kindle fonts folder might do the trick.

Anyway, I appreciate you looking into this.

[NiLuJe]

My bad, I wasn't precise enough. The *layout* engine doesn't do complex layout , even though it theoretically could in the case of the one used for KF8 files (FT+FC+Pango+Cairo).

EDIT: That's not entirely accurate, see Doitsu's original post, and my adventures below.
TL;DR: On a vanilla Kindle, Code2000 does complex layouts, but not Publisher Fonts.
On a hacked Kindle, it gets complicated, but you might be able to get away with complex layouts with a custom font.

So, yeah, it should mostly behave like a K3 on FW 3.4, with a wider unicode coverage, and possibly slightly less/different issues .

Fontconfig settings aren't usually terribly readable to begin with, but it's especially messy on the Kindle, so don't necessarily expect that what you see is what you'll actually get .

[NiLuJe]

Okay, first try broke the UI, yay ;D.

Second try, better, and seems to work. Will check that it doesn't sneakily break stuff.

Screenshots, for ref (w/ a custom FT, and on a Touch, for ease of recovery in case things go kablooey [and also because it's the one at the top of the pile ]).

The font used is SIL Scheherazade (which only ships a regular style, so the bold is synthetic, and happens to be a good testcase, since it renders Latin scripts horribly [w/ default ft settings (ie. bci, I suspect the autohinter would take better care of it]).

(First screenshot is w/ PubFont, second is Scheherazade [through fallback mechanism]).

EDIT: Nope, false alarm.

I'll merge this in the Fonts Hack once I've checked it doesn't break anything .

[Doitsu]

Originally Posted by NiLuJe

Okay, first try broke the UI, yay ;D.

I'm really sorry to hear this.

Originally Posted by NiLuJe

The font used is SIL Scheherazade (which only ships a regular style, so the bold is synthetic, and happens to be a good testcase, since it renders Latin scripts horribly [w/ default ft settings (ie. bci, I suspect the autohinter would take better care of it]).

The SIL Scheherazade screen capture looks great! Much better than the built-in code2000.ttf glyphs.

Originally Posted by NiLuJe

I'll merge this in the Fonts Hack once I've checked it doesn't break anything.

If it's not too much of a hassle, I'd really appreciate it, if you could create a separate fallback font replacement hack, because I'm quite happy with all the other system fonts and if I understand the way the font hack works correctly I'd need to provide replacement fonts for all system fonts regardless of whether I want to replace them or not.

I.e. in order to use the "regular font hack" I'd somehow have to get the default PW2 fonts, rename all of them and then "replace" existing fonts with identical fonts, which doesn't make much sense.

Again, thanks for your work!

[NiLuJe]

@Doitsu: The K5 version of the hack works very differently than on the K3 (it doesn't even ship with custom fonts), so you're free to cherry-pick exactly what you want to change, and it doesn't do anything by default, besides setting up the groundwork needed for the rest to function .

It's still not terribly user-friendly, but way less clunky, more flexible, and completely controlled through KUAL, so no need to play with specific files, or rename stuff.

I'll just add a dedicated entry to the "Font Override" menu .

[NiLuJe]

Ookay, back to the drawing board. Turns out, I'm blind, and unbreaking the UI actually broke something: it *looks* (can't read the script, so, guessing ^^) like the layout shaping is broken (like w/ PubFonts & actually selecting the custom font in the Aa menu)...

>_<"

EDIT²: ... Which is actually how Code2000 also renders on my device. Yay for custom FT tweaks. -_-".

EDIT³: For ref, how Scheherazade looks when that happens.

[NiLuJe]

Ungh! Got the pretty layout back.

It seems to inherit at least parts of the metrics of the font currently selected in the Aa menu, which might be fun on a PW >= 5.4.x, where the font actually isn't visible in the Aa menu.

Now to check on a PW, see if I can find something new to break ;p.

[NiLuJe]

Yup, terribly fun on a PW.

First screenshot is Scheherazade with Caecilia selected, second is "pure" Scheherazade, by sacrificing Baskerville. Notice the difference in line heights, and for some punctuation (?) marks [I hope that doesn't apply to diacritical marks than happen to be covered by the selected font...].

I'll check if the PW changes broke something on the Touch, and commit this.

On a sidenote, by virtue of how Amazon implemented Code2000, simply having the font in the cache is enough to get it picked up if it's the only one supporting a certain script. But I'd hit the layout issue with my custom FT (not sure how it would behave with Amazon's FT, but since PubFonts are 'broken' there too, I'm guessing the same).

[NiLuJe]

This has landed in the latest snapshots.

[poet1]

* Enter the font folder inside the folder you find linkfonts
Now let us font files to support Arabic and of course are 4 files and these are their names -
CJK.ttf
code2000.ttf
I18N.ttf
CREDITS.ttf
Ok what do you do now?
Simply choose one that suits you best calligraphy works and four copies of it and call them names of files in the above
And conveyed to the fonts folder and agree on a replacement, ended.
A very important piece of information: -
I experimented with some lines Unfortunately Matdam symbols in the book Vonsahkm handwriting times new roman Bold
The fact it was very cool and enjoy reading this line.

[NiLuJe]

@poet1: That vaguely looks like the folder structure of the K2/3/4 hack, so, err, a whole other kettle of fish .

(And that's on a K4B, right?)

EDIT: Actually not terribly sure how the KF8 renderer deals with Code2000 there... In Mobi7 files, it's replaced by I18N.ttf by the hack, but it doesn't do anything about Code2000 in KF8 files.
As on the K5, I *suspect* (but not quite sure, since Amazon seems to handle Code2000 differently there) just adding a family supporting an exotic script in linkfonts/fonts (w/ KF8 support enabled) will do the trick, with the same layout caveats than I was speaking of here when using a custom FT.
The KF8 handling is *very* messy in the K2/3/4 font hack, so it might be tricky to delve deeper.

[Doitsu]

@poet1What's the font shown in screen capture?
It would make a great replacement for the default font.