Word boundaries on Japanese text

[Kest]

Every time I convert a Japanese EPUB to AZW3, when I try to select a word on the book, Kindle can't detect the word boundaries and instead select the whole sentence (everything surrounded by dots or commas).

This only happens with my own converted books, the Japanese ones downloaded from Amazon gets the word boundaries without problems.

I've tried converting from Calibre, from Kindlegen, and even with a working book from amazon doing AZW3->EPUB->AZW3 cycle with KindleUnpack.

I'll like to ask if anyone had experienced the same issue or if anyone had got a converted Japanese book working on the Kindle without that problem.

The books are requesting the proper Japanese dictionary and showing the text vertically. Even the language of the book is set on Japanese (checked that on Calibre and with MobiMetaEditor), so I'm really clueless if this is and old issue or I'm doing something wrong.

[HarryT]

Questions about ebook creation belong in the "Kindle formats" forum, to where I'm now moving this thread.

[HarryT]

Have you compared the coding of a book that does work correctly with one that doesn't, to see what the differences are? Perhaps the Kindle requires some sort of non-printing word separator character in order to delineate words for dictionary lookup?

[KevinH]

Hi,
My guess is that that the Japanese language uses a different whitespace character than English does. I found this in a google search:

in Japanese, there is another space character, commonly called a 'full-width space'. According to my Mac's Character Viewer utility, this is U+3000 "IDEOGRAPHIC SPACE". This is (usually) what results when a user presses the space bar while typing in Japanese input mode.

So you could try a global find and replace for \u3000 to be a normal space " ", or you could try converting them to a numeric entity to see if that is really what is going on.

& # x 3 0 0 0 ;

In Sigil, you can edit your PreserveEntities Preferences in order to make these show up if that is what they are using.

Hope this helps,

KevinH

[Kest]

Japanese words usually are not separated by spaces or anything. It's true that there is a 'full-width space' when there is a space, but they are not used often, especially on books.

I'm not really sure how the Kindle detects the boundaries between Japanese words. I thought the Kindle itself with some help with dictionaries were in charge of that but i'm not sure anymore.

[HarryT]

That's why I suggested that you examine the coding of a Kindle book for which dictionary lookup does work, so you can see if there is any word delineation there.

[Kest]

Yes, sorry, I forgot to say that. I did look at the HTML code and there isn't any delimiter between words. Just plain text.

That at least on the html file extracted with KindleUnpack, but I doubt that kind of thing would be lost in that step

[DiapDealer]

Originally Posted by Kest

Yes, sorry, I forgot to say that. I did look at the HTML code and there isn't any delimiter between words. Just plain text.

That at least on the html file extracted with KindleUnpack, but I doubt that kind of thing would be lost in that step

Not lost, but quite possibly invisible. There are a host of zero-width characters that you'd never see by "looking at" the html code.

[Kest]

Ok, I did an exhaustive search with a hex editor on the main HTML file extracted from a working book from Amazon. Using an UTF8/hex table in some sentences I confirmed that indeed there are no hidden characters between words.

I doubt no one noticed this problem before if this is indeed a problem, so I'll greatly appreciate any past experiences from anyone converting Japanese text to Kindle.

[DiapDealer]

Have you tried using the DumpMobiHeader python script (included with the KindleUnpack archive) to compare the headers of a commercial (working) text and one you've converted yourself?

[Doitsu]

@Kest: IIRC, when Amazon introduced the Kindle in Japan they added XMDF support to KindleGen. Maybe all the books that work on your Kindle were converted from the XMDF format.
Try finding a free XMDF book on the Internet, unpack the book folder and convert it with KindleGen or create one from scratch following the instructions in Appendix D of the Kindle Publishing Guidelines.

It also couldn't hurt to also test the working Japanese books and your home-made books with the Kindle for iOS/Android/PCs apps to exclude eInk Kindle specific issues. (I'm assuming that you've installed the latest firmware and no homebrew apps that might interfere with the rendering of Japanese characters.)

Also remove all stylesheets and inline styles from your home-made books before you compile them with KindleGen to find out if this makes any difference. If it does, post the relevant stylesheets.

[nidl]

I have the same problem and still found no solution to it.

Surely the issue with the book formatting, but how to fix it or format properly I have no idea.

@Kest: did you find something else regarding this problem?

[yoshinama]

Sorry to revive this thread but I am facing the same issue and can't seem to find a solution. Has anyone managed to find a fix for this?

I can open a new thread if necroing is against the forum guidelines.

[colinsky]

Not sure if this will help you directly, but I have found Japanese word-boundary segmentation for dictionary lookup work very differently depending on the book format (MOBI, AZW, KFX). You might want to try converting a book to all three and see if any of them work in the way you'd like them too.

[jhowell]

I don’t think that MOBI supports Japanese.

KF8 (azw3) does but it relies on an included word boundary table (GESW records) that is generated during the publishing process. I do not think that there is any way to add that to a book that was not sold by Amazon.

I am not sure about KFX. It might work better in that format.

Update: Like KF8, KFX format also includes word boundary information, but only in published boooks.