Word boundaries on Japanese text

[Quoth]

Originally Posted by jhowell

I don’t think that MOBI supports Japanese.

The old mobi/azw/KF7 only supports a subset of Greek in addition to Latin-Roman. Lack of fonts on the ereaders is one issue.
No Chinese, Korean, Hindi, Japanese (or any Asian).
No Cyrllic, Hebrew or Arabic.
The fonts installed do support Icelandic, Spanish, French, German etc extra letters not used in English.

It was really very bad of Amazon to release such a limited GUI/Language support in 2007. The underlying OS supported it over 10 years earlier. They seem to just bolt the Mobipocket stuff they bought in 2005 on top of Linux. It was very limited as it had originally supported Palm OS, Symbian, Windows CE and maybe DOS and very limited.

[colinsky]

Originally Posted by jhowell

I don’t think that MOBI supports Japanese.

KF8 (azw3) does but it relies on an included word boundary table (GESW records) that is generated during the publishing process. I do not think that there is any way to add that to a book that was not sold by Amazon.

I am not sure about KFX. It might work better in that format.

Update: Like KF8, KFX format also includes word boundary information, but only in published boooks.

Thanks. Most everything I have played with has been from an original EPUB source (converted by Calibre) so I'm not sure how the behavior I am seeing is arising.

Basically: MOBI doesn't seem to attempt any word segmentation (you can select any combination of characters. AZW/KFX only let you select along some notion of word boundaries (kanji + inflection). I'll have to do some more experimentation, to see if it is deriving from a source feature (like a RUBY tag) or some other process.

Are you aware of documentation (or reverse engineering) that describes the word boundary information in KF8 or KFX?

[jhowell]

Originally Posted by colinsky

Are you aware of documentation (or reverse engineering) that describes the word boundary information in KF8 or KFX?

I have not seen any documentation on this. I looked into the KF8 GESW years ago but did not write up my findings. If I recall correctly it is a compressed table of coded instructions for parsing the raw HTML content to determine which bytes make up each word, taking into account that they might not be contiguous due to HTML markup.