Understanding bidirectional Unicode control characters

[PoP]

When "studying" Unicode specifications I came across Carl Henderson's remarkably clear and concise illustration of understanding bidirectional text.

I extracted the .html from his page and I created a .mobi and a .azw3 to test the capabilities of my Kindle Keyboard v3.4. The expected behaviour is explained in the text along with coding and rendering examples, greatly enhancing the comprehension.

The subject is relatively complex and certainly not trivial. (I think) I *do* understand it now (?)... So can you! Hoping you find it useful to test other browsers and ereaders.

I have identified the following rendering errors (I think) . And I would appreciate if a dedicated and applied reader interested in this subject could confirm my understanding.

When Adobe Acrobat XI Pro v11.0, Firefox v20.0.1 or Microsoft Internet Explorer v10.0 render the .html
The "implicit marker, third case"

Can all these good software be doing that same error?!

After converting the .html to .mobi with Kindle Previewer v2.85, to .azw3 with Calibre 0.9.27 and viewing them in my Kindle Keyboard v3.4
The "implicit marker, first case"


Open my test files below to better see what I mean:

[Doitsu]

While Unicode experts love to discuss these issues, in real life, there are hardly any problems, as long as you use the dir="rtl" attribute for predominantly RTL text. And even if you don't do this, RTL text will be mostly displayed correctly, if it doesn't contain embedded LTR text, except for final punctuation characters at the end of a paragraph, which will always be displayed on the right hand side, even if they should be displayed after the last word of the sentence on the light hand side.
Also most rendering engines automatically use RTL rendering & shaping, when they encounter Unicode glyphs that need to be rendered RTL and switch to LTR when they encounter Latin text. I.e. most of these fancy RTL/LTR embedding codes aren't really necessary in real life applications. (I.e. mostly RTL text with LTR segments and vice versa.) Also some of them aren't fully supported by rendering engines anyway.

Please find attached a simple test file that I just slapped together using a BBC Arabic article about the late Margret Thatcher. On the first HTML page I inserted her English name in an Arabic paragraph, and on the third page I inserted her (transliterated) Arabic name in an English paragraph. Both inserts where not enclosed by any special RTL or LTR embedding codes.

(The second HTML page shows what happens if you don't use the dir="rtl" attribute: punctuation characters are always displayed on the right hand side and inserted LTR text breaks up the paragraph.)

tl;dr RTL text rendering isn't as complicated as it looks like

[PoP]

So true. Thanks for your explanations and real life examples. Your wise advice (and the rendering errors I pointed to) show how to stay out of trouble.

I am still just an apprentice, but it was dnatsrednu ot nuf hcum! admittedly spending far too much time on the theory.

Again, sincere thanks for sharing your expertise.

[neufsix]

Thank you! I planned to work on a book mixing latin and hebrew charaters, it comes in handy.

[HarryT]

Does ADE support RTL languages these days? ISTR that earlier versions of it didn't.

[Doitsu]

Originally Posted by HarryT

Does ADE support RTL languages these days? ISTR that earlier versions of it didn't.

AFAIK, the latest ADE version still doesn't support Arabic. However, it's possible to display Arabic text (without vowel signs) with ADE, if you're willing to jump through some hoops.

1. You'll obviously need to embed an Arabic font
2. You'll need to manually generate the Arabic context forms.
3. You'll need to reverse the order of the preshaped Arabic letters.

(The last two steps can be accomplished with this free tool.)

Since all of this is a major PITA, it's only practical for very short quotes, but it can be done.

Please find attached an excerpt from a German book available at MR that demonstrates this technique.

[HarryT]

Ouch - that does sound painful. And presumably, if you reverse the order of the letters, and then do a conversion to a format which does support RTL, such as KF8, you'd need to reverse the letters again to make it display correctly!

[Doitsu]

Originally Posted by HarryT

Ouch - that does sound painful. And presumably, if you reverse the order of the letters, and then do a conversion to a format which does support RTL, such as KF8, you'd need to reverse the letters again to make it display correctly!

Actually, that part isn't so complicated. You can use this clever method that Jellby came up with to test for ADE readers and have all non-ADE readers display regular Arabic text.

(Note the additional vowel signs on the first Arabic word in the screenshot of the Kindle version, which was generated without further modification from the .epub file attached to my previous post.)

[HarryT]

Very cunning - thanks!

[PoP]

Originally Posted by Doitsu

AFAIK, the latest ADE version still doesn't support Arabic. However, it's possible to display Arabic text (without vowel signs) with ADE, if you're willing to jump through some hoops.

1. You'll obviously need to embed an Arabic font
2. You'll need to manually generate the Arabic context forms.
3. You'll need to reverse the order of the preshaped Arabic letters.

(The last two steps can be accomplished with this free tool.)

Since all of this is a major PITA, it's only practical for very short quotes, but it can be done.

Please find attached an excerpt from a German book available at MR that demonstrates this technique.

While playing some more with text transformations:

Spoiler:

<?xml version='1.0' encoding='utf-8'?>
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <style> 

div
{
width:500px;
height:75px;
border:1px solid black;
text-align:center;
}

.rx
{
transform:rotateX(180deg);
-webkit-transform:rotateX(180deg);
}

.ry
{
transform:rotateY(180deg);
-webkit-transform:rotateY(180deg);
}

.rz
{
transform: rotate(180deg);
-webkit-transform:rotate(180deg);
}

.rx120
{
transform:rotateX(120deg);
-webkit-transform:rotateX(120deg);
}

</style>
    <title>CSS3 &amp; other transforms</title>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
  </head>
  <body>

<p>no transform : 
</p><div>© Copyleft Creative Common Share Alike License</div><p></p>

<p>3D -webkit-transform:rotateX(180deg);
</p><div class="rx">© Copyleft Creative Common Share Alike License</div><p></p>

<p>3D -webkit-transform:rotateY(180deg);
</p><div class="ry">© Copyleft Creative Common Share Alike License</div><p></p>

<p>2D -webkit-transform:rotate(180deg);
</p><div class="rz">© Copyleft Creative Common Share Alike License</div><p></p>

<p>3D -webkit-transform:rotateX(180deg); 3D -webkit-transform:rotateY(180deg);
</p><div class="rx"><div class="ry">© Copyleft Creative Common Share Alike License</div></div><p></p>

<p>3D -webkit-transform:rotateX(180deg); 2D -webkit-transform:rotate(180deg);
</p><div class="rx"><div class="rz">© Copyleft Creative Common Share Alike License</div></div><p></p>

<p>3D -webkit-transform:rotateY(180deg); 2D -webkit-transform:rotate(180deg);
</p><div class="ry"><div class="rz">© Copyleft Creative Common Share Alike License</div></div><p></p>

<p>3D -webkit-transform:rotateX(180deg); 3D -webkit-transform:rotateY(180deg); 2D -webkit-transform:rotate(180deg);
</p><div class="rx"><div class="ry"><div class="rz">© Copyleft Creative Common Share Alike License</div></div></div><p></p>

<p>&lt;bdo dir="rtl"&gt; </p><div><bdo dir="rtl">© Copyleft Creative Common Share Alike License</bdo></div><p></p>

<p>3D -webkit-transform:rotateX(120deg);
</p><div class="rx120">© Copyleft Creative Common Share Alike License</div><p></p>


</body></html>

I thought that the <bdo dir="rtl"> tag in my example #9 could possibly ease step 3.


Though I still prefer the elegance of simple Unicode ʇɟǝןʎdoƆ (ↄ) for reversing (c) Copyright.