Japanese characters not showing up on some devices

[shibuyaloren]

Hi everyone, I'm a newbie to both eBook formatting and Calibre, but willing to learn!

I have an ePub novel that was originally typeset in Reedsy. There are several instances of Japanese kanji characters in the book that show up fine in Calibre and in Kindle Previewer. However, when I send it over to my Kobo Libre, the Japanese text is deleted out. (Doesn't display as a blank or as weird characters; just vanishes entirely.) I know my Kobo has the ability to display kanji, because I just read a book that had it, and it was definitely text and not an image of the characters dropped in. Something in this book is not set up right, I'm guessing, but I'm not positive what. (I also tried generating ePubs from LibreOffice and Scrivener and they won't display the kanji either, so it's not a problem unique to the Reedsy-typeset file.)

I was trying to follow directions to embed the fonts after seeing older posts in this forum, thinking that might be a fix, but I've never worked in-depth with Calibre yet and I don't seem to be doing it right:

Spoiler:

Can someone nudge me in the right direction?

Thanks!

[salamanderjuice]

I think you need to embed a font that has CJK character support like Noto. The fonts shown embedded there don't cover Japanese or Chinese characters.

[Notjohn]

If you hope to sell this book, be aware that embedding fonts can lead to trouble on Amazon's KDP. Also note that Calibre, while a wonderful library management tool, is not the best way to format a book for sale. For that, Sigil is the go-to software.

[Tex2002ans]

Originally Posted by shibuyaloren

Hi everyone, I'm a newbie to both eBook formatting and Calibre, but willing to learn!

Fantastic. Welcome to MobileRead!

Originally Posted by shibuyaloren

There are several instances of Japanese kanji characters in the book that show up fine in Calibre and in Kindle Previewer. However, when I send it over to my Kobo Libre, the Japanese text is deleted out. (Doesn't display as a blank or as weird characters; just vanishes entirely.)

[...]

I was trying to follow directions to embed the fonts after seeing older posts in this forum, thinking that might be a fix, [...].

Can someone nudge me in the right direction?

Do I have the thread for you:

2020: "Should Chinese Fonts be Embedded in Ebooks?"

I was working on a few journal articles that had a handful of Chinese/Japanese characters.

The topic goes into a lot of technical discussion though, so I'll try to create a more easy tutorial here.

* * *

This is what you'll do:

1. Use Proper HTML Language in Your Document

Add this code around all your Japanese text:

<span class="japanese" lang="ja" xml:lang="ja">返</span>

Don't be scared. Let me break this down into its 3 main pieces:

• <span class="japanese">
  
  • Helps you apply CSS + fonts later.
• lang="ja" xml:lang="ja"
  
  • Says "Hey! I'm Japanese text!"
    
  • Note: Every language uses different 2-letter codes. So you'll need to change if you're doing Korean ("ko") or Chinese ("zh") or Vietnamese ("vi").
• 返
  
  • Where you put your actual "foreign" characters.

2. Find Font That Includes Characters You Need

A big list of CJK fonts can be found on Wikipedia:

https://en.wikipedia.org/wiki/List_of_CJK_fonts

For example, the "Source Han" or "Noto" fonts have a license that allows you to embed them in ebooks for free.

3. Download the OTF or TTF

Once you've chosen your font, download the OTF or TTF versions. (This is the format needed for EPUBs.)

For example, here is the latest page for "Source Han Sans":

https://github.com/adobe-fonts/sourc...ses/tag/2.004R

(As of today, it's v2.004. Yes, fonts have version numbers + they get updated/fixed!)

Every font will be slightly different in organization/naming conventions...

But in this case, you want to download:

• Static Language Specific OTFs
  
  • Note: This will be a huge 765 MB file.

Unzip the file, and you'll see multiple folders. What you want is buried in OTF:

- SourceHanSans
-- OTF
--- Japanese
---- SourceHanSans-Regular.otf

You can install that font to your computer and/or you can use that file to shove into your ebook.

4. Add the CSS

Now, you want to go back into your EPUB.

This time, we'll adjust the CSS.

In your CSS file, add this:

span.japanese {
	font-family: "Source Han Sans", sans-serif;
}

Remember when we did the <span class="japanese"> code above?

This says to use the font ONLY for those specific pieces.

And in plain English, the code says:

"For every <span> that has a class named 'japanese': Use the font Source Han Sans. If you can't find that, use the device's default sans-serif font."

5. Insert Font Into Your EPUB

5.1. If you're using Calibre:

• Tools > Embed Referenced Fonts

or you can:

• File > Import Files into Book
  
  • Find + add the font's OTF file.

5.2. If you're using Sigil:

• File > Existing File
  
  • Find + add the font's OTF file.
• Use Doitsu's "CSSFromFonts" plugin.

5.3. Or you can manually put this at the top of your CSS:

@font-face {
	font-family: "Source Han Sans";
	font-weight: normal;
	font-style: normal;
	src: url(“../Fonts/SourceHanSans-Regular.otf”);
}

In plain English, this says:

"I have a new font here! The font's name is Source Han Sans. It is not bold and not italics. This is the location of the font file."

(Optional) 6. Subset your Fonts

If Calibre: Tools > Subset Embedded Fonts
If Sigil: Use Doitsu's "SubsetFonts" plugin.

This will cut down the filesize.

Asian font files are huge, because there are thousands and thousands of characters.

If you're only using a handful (like in my journal articles, there were maybe ~20 characters total), you can shave the font from many MBs down to a few hundred KBs.

In plain English:

What does subsetting do exactly?

Subsetting looks through your specific book + font, then deletes every letter you're not using.

Imagine you only used the fancy font on your book's title page: "I CUP".

You can remove all those thousands and thousands of characters, only leaving 4 within the subsetted font: I + C + U + P!

Originally Posted by Notjohn

Also note that Calibre, while a wonderful library management tool, is not the best way to format a book for sale. For that, Sigil is the go-to software.

Complete hogwash. Both are perfectly capable EPUB editors.

Calibre's Editor does some things better than Sigil, like External Link check + Insert Special Character.

And Sigil does some things better than Calibre, like easier TOC editing + easier-to-read Reports.

[Quoth]

Excellent tutorial.

And then even when done properly it won't work on many older ereaders, because the companies took lazy Western-centric view. The actual OSes used had Cyrillic, Arabic, Chinese, Japanese, Thai, Hindi, Hebrew etc over a decade before dedicated ebook readers were developed. Amazon Kindle was one of the most backward.

Both Sigil and Calibre can be better for ebooks than expensive Indesign as it's fudged for ebooks, it was developed for PDFs and production of papers, magazines and technical books. Even for PDF novels a free Word processor can be better than InDesign.

[Tex2002ans]

Originally Posted by Quoth

Excellent tutorial.

Originally Posted by Quoth

And then even when done properly it won't work on many older ereaders, because the companies took lazy Western-centric view. The actual OSes used had Cyrillic, Arabic, Chinese, Japanese, Thai, Hindi, Hebrew etc over a decade before dedicated ebook readers were developed. Amazon Kindle was one of the most backward.

Text Encoding, Internationalization, and layout (for Chinese + Arabic + RTL languages) are all complicated beasts.

Check out these two fantastic videos by Computerphile (especially the first one!):

"Internationalis(z)ing Code"

"Characters, Symbols and the Unicode Miracle"

And many other details were discussed in that "Should Chinese be Embedded" thread above.

If you're interested in more, check out the Harfbuzz + LibreOffice talks I linked to in Post #8.

Harbuzz is the renderer that figures out how to actually draw the characters (it's now the basis of many programs/browsers).

And the LibreOffice talks were some of the Asian users discussing common bugs/problems that crop up in programs like Word/LibreOffice. Plus needing to keep in mind special cases, like how they input characters using IME.

[shibuyaloren]

Thank you so much! Luckily, the Kindle version displays properly and the typesetting looks great right out of the box, it's the other readers I'm trying to tackle - but I'll definitely have a closer look at Sigil once I get used to Calibre and CSS adjustments.

I really appreciate the tutorial - will get started on how to apply it tomorrow!

[Tex2002ans]

Originally Posted by shibuyaloren

Luckily, the Kindle version displays properly and the typesetting looks great right out of the box, [...]

Not necessarily.

Kindle Previewer 3 only shows you the latest devices/formats...

But you still have to keep in mind fallback code for very old Kindle devices that only read the original MOBI format (like the Kindle DX).

Sadly, Kindle Previewer 3 doesn't show these ol' devices in the dropdown anymore. You'd need to run an old version of Kindle Previewer 2 (or have a very old device on hand).

* * *

Can you show more actual examples of the Japanese usage in this book?

In the case of my journal articles, they all included full English transliterations right next to the original Chinese/Japanese words:

Liu E, also known as Liu Tieyun <span class="japanese" lang="ja" xml:lang="ja">劉鐵雲</span>, was born in 1857 at Liuhe <span class="japanese" lang="ja" xml:lang="ja">六合</span> county in what is today Nanjing <span class="japanese" lang="ja" xml:lang="ja">南京</span>.

so for the old MOBI fallback, an "okay" solution was in Post #25 of the embedded thread:

@media amzn-mobi {
	.chinese {
		display: none;
	}
	.japanese {
		display: none;
	}
}

What this code says is:

- If the format is Amazon's old MOBI format.
- And the class is "chinese" or "japanese"
- Hide it.

Instead of a reader seeing "missing boxes"... it'll just be disappeared, and all you'd see is the English text:

Liu E, also known as Liu Tieyun, was born in 1857 at Liuhe county in what is today Nanjing.

Originally Posted by shibuyaloren

[...] it's the other readers I'm trying to tackle - but I'll definitely have a closer look at Sigil once I get used to Calibre and CSS adjustments.

Another trick that'll help.

As you're following Step 1, adding the proper HTML lang markup, you can use Calibre's/Sigil's fantastic Spellcheck Lists.

In Calibre: Tools > Check Spelling.

In Sigil: Tools > Spellcheck > Spellcheck.

You can then sort by the "Word" or "Language" column.

Here's what it looks like in Sigil 1.5.1:



This lets you easily spot words you haven't marked yet or accidentally marked wrongly.

Like you might see a Japanese word that says Language: "English".

You can then double-click on the word to jump to its location in the ebook, then add that Step 1 <span> code around it. Now next time you refresh the Spellcheck List, bam, it'll say Japanese!

[Hitch]

Originally Posted by Tex2002ans

Not necessarily.

Kindle Previewer 3 only shows you the latest devices/formats...

But you still have to keep in mind fallback code for very old Kindle devices that only read the original MOBI format (like the Kindle DX).

Sadly, Kindle Previewer 3 doesn't show these ol' devices in the dropdown anymore. You'd need to run an old version of Kindle Previewer 2 (or have a very old device on hand).

To this day, when we create eBooks with Kanji, (Juas as when we use Cambria Math, for that matter), we create two 'sets' of books for KDP. The KF8-KFX version, which uses the embedded fonts and the KF7-ish version, which instead uses images, managed with media-queries.

There really isn't a choice, at this point in time. You cannot upload a book at Amazon that says "Hey, I'm kF8/KFX only, don't sell me to people with Kindle Keyboards," so...you have to take the longer view.

(n.b.: well, you could upload a book that says KF8/KFX only, by using fixed-layout but we don't do that for all the obvious reasons.)

Hitch

[DNSB]

Looking at the OP's message, he references a Kobo Libre. The Libre has a couple of CJK fonts available but you will need to select one of them which will set also set that font as the default for the next book. You also need to avoid using font-family references in the CSS which can block the ability to select either Tsukushi Mincho or UD Kakugo as the display font when displaying Japanese characters when creating an epub that will be displayed with the RMSDK renderer. Creating a kepub will use Kobo's epub3 renderer which will search for a glyph that is not defined in the current font. At time the results of doing this can be a bit strange.

[JSWolf]

Originally Posted by Notjohn

If you hope to sell this book, be aware that embedding fonts can lead to trouble on Amazon's KDP. Also note that Calibre, while a wonderful library management tool, is not the best way to format a book for sale. For that, Sigil is the go-to software.

Calibre works perfectly well to format an eBook. I do use it. No slight against Sigil. I just want to correct your incorrect statement.

[JSWolf]

Originally Posted by Hitch

There really isn't a choice, at this point in time. You cannot upload a book at Amazon that says "Hey, I'm kF8/KFX only, don't sell me to people with Kindle Keyboards," so...you have to take the longer view.

I thought for sure you'd know know that the K3/KK has support for KF8.

[Tex2002ans]

Originally Posted by Hitch

To this day, when we create eBooks with Kanji, [...] we create two 'sets' of books for KDP. The KF8-KFX version, which uses the embedded fonts and the KF7-ish version, which instead uses images, managed with media-queries.

Thanks for the info.

Have any insights into mass generating the images? And/or resizing them correctly to the text reliably?

I think I have a few workflow ideas with ImageMagick (similar to my formulas generation), but that's a whole other thing.

And I personally haven't messed with inline images too much since way back during that SVG/apple thread.

* * *

Side Note: For book club, I've recently been reading Jordan Peterson's latest book:

"Beyond Order: 12 More Rules for Life", published by Penguin Random House.

The EPUB had two Hebrew words in there as inline images:

[...] translation of a biblical Hebrew phrase, <i>tohu wa-bohu</i> (<img alt="בֹ֔הוּ" class="h1em-HEB" src="../images/page_257.jpg"/>), made up of two words, <i>tohuw</i> and <i>bohuw</i>.

[...]

It is also associated with another Hebrew word, <i>tehom</i> (<img alt="תּ֔ הוֹם" class="h1em-HEB" src="../images/page_258.jpg"/>), which is the source of the phrase “the deep.”

with this CSS:

img.h1em-HEB {
	height: 1em;
	vertical-align: bottom;
}

it looks like it worked on my Android phone using "PocketBook Reader"... but I haven't thoroughly tested elsewhere.

And I'm glad to see they used the proper alt tags!

[Quoth]

Originally Posted by JSWolf

I thought for sure you'd know know that the K3/KK has support for KF8.

Maybe not originally, but certainly current FW works KK3 (B008), but not the later B009 DXG because it's really a DX (pre KK3) with upgraded screen. It's terrible for even Mobi Font support compared to KK3 with old KF7 mobi. Perhaps lacked RAM. The original DX was too poor in SW and RAM etc for the intended Academic market. Hence unsold DXGs firesaled even after PW3 came out!
I test any new styles on a DXG and KK3 (using Dual Mobi and also Publisher option and default on the KK3).

[JSWolf]

Originally Posted by Quoth

Maybe not originally, but certainly current FW works KK3 (B008), but not the later B009 DXG because it's really a DX (pre KK3) with upgraded screen. It's terrible for even Mobi Font support compared to KK3 with old KF7 mobi. Perhaps lacked RAM. The original DX was too poor in SW and RAM etc for the intended Academic market. Hence unsold DXGs firesaled even after PW3 came out!
I test any new styles on a DXG and KK3 (using Dual Mobi and also Publisher option and default on the KK3).

Where did I mention the DX/G?