Japanese characters not showing up on some devices
 

Hi everyone, I'm a newbie to both eBook formatting and Calibre, but willing to learn!

I have an ePub novel that was originally typeset in Reedsy. There are several instances of Japanese kanji characters in the book that show up fine in Calibre and in Kindle Previewer. However, when I send it over to my Kobo Libre, the Japanese text is deleted out. (Doesn't display as a blank or as weird characters; just vanishes entirely.) I know my Kobo has the ability to display kanji, because I just read a book that had it, and it was definitely text and not an image of the characters dropped in. Something in this book is not set up right, I'm guessing, but I'm not positive what. (I also tried generating ePubs from LibreOffice and Scrivener and they won't display the kanji either, so it's not a problem unique to the Reedsy-typeset file.)

I was trying to follow directions to embed the fonts after seeing older posts in this forum, thinking that might be a fix, but I've never worked in-depth with Calibre yet and I don't seem to be doing it right:



Can someone nudge me in the right direction?

Thanks!

I think you need to embed a font that has CJK character support like Noto. The fonts shown embedded there don't cover Japanese or Chinese characters.

If you hope to sell this book, be aware that embedding fonts can lead to trouble on Amazon's KDP. Also note that Calibre, while a wonderful library management tool, is not the best way to format a book for sale. For that, Sigil is the go-to software.

Fantastic. Welcome to MobileRead! :)

Do I have the thread for you:

2020: "Should Chinese Fonts be Embedded in Ebooks?"

I was working on a few journal articles that had a handful of Chinese/Japanese characters.

The topic goes into a lot of technical discussion though, so I'll try to create a more easy tutorial here. :D

* * *

This is what you'll do:

1. Use Proper HTML Language in Your Document

Add this code around all your Japanese text:

Don't be scared. Let me break this down into its 3 main pieces:

• <span class="japanese">
  
  • Helps you apply CSS + fonts later.
• lang="ja" xml:lang="ja"
  
  • Says "Hey! I'm Japanese text!"
    
  • Note: Every language uses different 2-letter codes. So you'll need to change if you're doing Korean ("ko") or Chinese ("zh") or Vietnamese ("vi").
• 返
  
  • Where you put your actual "foreign" characters.

2. Find Font That Includes Characters You Need

A big list of CJK fonts can be found on Wikipedia:

https://en.wikipedia.org/wiki/List_of_CJK_fonts

For example, the "Source Han" or "Noto" fonts have a license that allows you to embed them in ebooks for free.

3. Download the OTF or TTF

Once you've chosen your font, download the OTF or TTF versions. (This is the format needed for EPUBs.)

For example, here is the latest page for "Source Han Sans":

https://github.com/adobe-fonts/sourc...ses/tag/2.004R

(As of today, it's v2.004. Yes, fonts have version numbers + they get updated/fixed!)

Every font will be slightly different in organization/naming conventions...

But in this case, you want to download:

• Static Language Specific OTFs
  
  • Note: This will be a huge 765 MB file.

Unzip the file, and you'll see multiple folders. What you want is buried in OTF:

- SourceHanSans
-- OTF
--- Japanese
---- SourceHanSans-Regular.otf

You can install that font to your computer and/or you can use that file to shove into your ebook.

4. Add the CSS

Now, you want to go back into your EPUB.

This time, we'll adjust the CSS.

In your CSS file, add this:

Remember when we did the <span class="japanese"> code above?

This says to use the font ONLY for those specific pieces. :)

And in plain English, the code says:

"For every <span> that has a class named 'japanese': Use the font Source Han Sans. If you can't find that, use the device's default sans-serif font."

5. Insert Font Into Your EPUB

5.1. If you're using Calibre:

• Tools > Embed Referenced Fonts

or you can:

• File > Import Files into Book
  
  • Find + add the font's OTF file.

5.2. If you're using Sigil:

• File > Existing File
  
  • Find + add the font's OTF file.
• Use Doitsu's "CSSFromFonts" plugin.

5.3. Or you can manually put this at the top of your CSS:

In plain English, this says:

"I have a new font here! The font's name is Source Han Sans. It is not bold and not italics. This is the location of the font file."

(Optional) 6. Subset your Fonts

If Calibre: Tools > Subset Embedded Fonts
If Sigil: Use Doitsu's "SubsetFonts" plugin.

This will cut down the filesize.

Asian font files are huge, because there are thousands and thousands of characters.

If you're only using a handful (like in my journal articles, there were maybe ~20 characters total), you can shave the font from many MBs down to a few hundred KBs.

In plain English:

What does subsetting do exactly?

Subsetting looks through your specific book + font, then deletes every letter you're not using.

Imagine you only used the fancy font on your book's title page: "I CUP".

You can remove all those thousands and thousands of characters, only leaving 4 within the subsetted font: I + C + U + P! :)

Complete hogwash. Both are perfectly capable EPUB editors.

Calibre's Editor does some things better than Sigil, like External Link check + Insert Special Character.

And Sigil does some things better than Calibre, like easier TOC editing + easier-to-read Reports.

Excellent tutorial.

And then even when done properly it won't work on many older ereaders, because the companies took lazy Western-centric view. The actual OSes used had Cyrillic, Arabic, Chinese, Japanese, Thai, Hindi, Hebrew etc over a decade before dedicated ebook readers were developed. Amazon Kindle was one of the most backward.

Both Sigil and Calibre can be better for ebooks than expensive Indesign as it's fudged for ebooks, it was developed for PDFs and production of papers, magazines and technical books. Even for PDF novels a free Word processor can be better than InDesign.

:thumbsup: :thumbsup:

Text Encoding, Internationalization, and layout (for Chinese + Arabic + RTL languages) are all complicated beasts.

Check out these two fantastic videos by Computerphile (especially the first one!):

"Internationalis(z)ing Code"

"Characters, Symbols and the Unicode Miracle"

And many other details were discussed in that "Should Chinese be Embedded" thread above.

If you're interested in more, check out the Harfbuzz + LibreOffice talks I linked to in Post #8.

Harbuzz is the renderer that figures out how to actually draw the characters (it's now the basis of many programs/browsers).

And the LibreOffice talks were some of the Asian users discussing common bugs/problems that crop up in programs like Word/LibreOffice. Plus needing to keep in mind special cases, like how they input characters using IME.

Thank you so much! Luckily, the Kindle version displays properly and the typesetting looks great right out of the box, it's the other readers I'm trying to tackle - but I'll definitely have a closer look at Sigil once I get used to Calibre and CSS adjustments.

I really appreciate the tutorial - will get started on how to apply it tomorrow!

Not necessarily.

Kindle Previewer 3 only shows you the latest devices/formats...

But you still have to keep in mind fallback code for very old Kindle devices that only read the original MOBI format (like the Kindle DX).

Sadly, Kindle Previewer 3 doesn't show these ol' devices in the dropdown anymore. You'd need to run an old version of Kindle Previewer 2 (or have a very old device on hand).

* * *

Can you show more actual examples of the Japanese usage in this book?

In the case of my journal articles, they all included full English transliterations right next to the original Chinese/Japanese words:

so for the old MOBI fallback, an "okay" solution was in Post #25 of the embedded thread:

What this code says is:

- If the format is Amazon's old MOBI format.
- And the class is "chinese" or "japanese"
- Hide it.

Instead of a reader seeing "missing boxes"... it'll just be disappeared, and all you'd see is the English text:

Another trick that'll help.

As you're following Step 1, adding the proper HTML lang markup, you can use Calibre's/Sigil's fantastic Spellcheck Lists.

In Calibre: Tools > Check Spelling.

In Sigil: Tools > Spellcheck > Spellcheck.

You can then sort by the "Word" or "Language" column.

Here's what it looks like in Sigil 1.5.1:

Attachment 187303

This lets you easily spot words you haven't marked yet or accidentally marked wrongly.

Like you might see a Japanese word that says Language: "English".

You can then double-click on the word to jump to its location in the ebook, then add that Step 1 <span> code around it. :) Now next time you refresh the Spellcheck List, bam, it'll say Japanese!

To this day, when we create eBooks with Kanji, (Juas as when we use Cambria Math, for that matter), we create two 'sets' of books for KDP. The KF8-KFX version, which uses the embedded fonts and the KF7-ish version, which instead uses images, managed with media-queries.

There really isn't a choice, at this point in time. You cannot upload a book at Amazon that says "Hey, I'm kF8/KFX only, don't sell me to people with Kindle Keyboards," so...you have to take the longer view.

(n.b.: well, you could upload a book that says KF8/KFX only, by using fixed-layout but we don't do that for all the obvious reasons.)

Hitch

Looking at the OP's message, he references a Kobo Libre. The Libre has a couple of CJK fonts available but you will need to select one of them which will set also set that font as the default for the next book. You also need to avoid using font-family references in the CSS which can block the ability to select either Tsukushi Mincho or UD Kakugo as the display font when displaying Japanese characters when creating an epub that will be displayed with the RMSDK renderer. Creating a kepub will use Kobo's epub3 renderer which will search for a glyph that is not defined in the current font. At time the results of doing this can be a bit strange.

Calibre works perfectly well to format an eBook. I do use it. No slight against Sigil. I just want to correct your incorrect statement.

:smack: I thought for sure you'd know know that the K3/KK has support for KF8.

Thanks for the info.

Have any insights into mass generating the images? And/or resizing them correctly to the text reliably?

I think I have a few workflow ideas with ImageMagick (similar to my formulas generation), but that's a whole other thing. :D

And I personally haven't messed with inline images too much since way back during that SVG/apple thread.

* * *

Side Note: For book club, I've recently been reading Jordan Peterson's latest book:

"Beyond Order: 12 More Rules for Life", published by Penguin Random House.

The EPUB had two Hebrew words in there as inline images:

with this CSS:

it looks like it worked on my Android phone using "PocketBook Reader"... but I haven't thoroughly tested elsewhere.

And I'm glad to see they used the proper alt tags! :)

Maybe not originally, but certainly current FW works KK3 (B008), but not the later B009 DXG because it's really a DX (pre KK3) with upgraded screen. It's terrible for even Mobi Font support compared to KK3 with old KF7 mobi. Perhaps lacked RAM. The original DX was too poor in SW and RAM etc for the intended Academic market. Hence unsold DXGs firesaled even after PW3 came out!
I test any new styles on a DXG and KK3 (using Dual Mobi and also Publisher option and default on the KK3).

Where did I mention the DX/G?

I am, in fact, working on something perhaps-new right this moment, for a book with many hundreds of non-unicode equations. (sigh). SVG in KF8 has been problematic, for some time, other than for the covers and even then, it's problematic.

And here I keep waiting for YOU to come up with The Answer, bud!

* * *

Whut? they identified images as Hebrew? WHY?

Yes, but why? For images????

Hitch

Almost exactly like this Japanese case... fonts missing Hebrew letters.

Plus who knows how a device might f-up rendering RTL languages within LTR text... especially RTL within an LTR sentence!

(Again, check out the fantastic Internationalization video + Harfbuzz talk on Arabic/Hebrew/East-Asian font rendering.)

Of course, I'd prefer the actual Unicode Hebrew within my EPUBs... but it was an "okay" solution. (And using proper alt means TTS + you can substitute or regenerate in the future!)

* * *

Actually, I think they may have botched the ebook slightly.

Originals:

Attachment 187339
Attachment 187340

LibreOffice Writer (v7.1.3.2):

Attachment 187341

1st word's alt text is definitely missing the 2nd chunk.

In 2nd example, LibreOffice has 2 dots above. In original image, it's two dots below.

Unsure which rendering is correct though, since I don't read Hebrew. :D

lol, I know you have the solution. Just spread the knowledge and tell me!!! :D

Closest thing I ever come across in all my books is Polytonic Greek. (Written about many times over the years.)

And luckily, nearly all inline equations I come across can all be converted to simple form. So something like this:

can be converted to:

Uh oh, another Thermodynamics book?

?? Maybe I misread your post, but didn't you post code that simply displayed an image of the text? Did I misread it?


Would I do that to you?????? :D

Hitch

Yes. Displayed (Hebrew) text as an inline image.

Isn't that what we were talking about? :blink:

You mentioned Kanji = two versions: 1 embedded fonts + 1 media-query images.

I mentioned latest book I've been reading: inline Hebrew images.

Well, it was definitely fun pushing the boundaries. Pushed me to learn a ton more about Maths/Equation typesetting as well. :)

And you can't just go dangling the word "equations" out there and not let me know at least a little bit of details! I must know!

I see. You were discussing it as if it also had text. Gotcha. Yes, that's what we do for Kanji (or anything else that has MQs with text in one "version" and images in the other); wrap them in lang tags and pray a lot.



Honestly, it would take a LOT to convince me to ever do that again. Noit to mention that effectively, between the line-edit, etc., we worked on it for over a year. Ugh.

(This is a book on statistics and the bigger issue are what I refer to as b-hats, or hatted-bs. Hatted-S'es have a unicode character, but hatted-Bs do not, so we'll have to do all the hatted-bs, and all the fraction equations, as images. [sigh].) And you know, inline SVG in "Kindle" (aka, MOBI or ePUB or whatever damned format) doesn't work right. Bugger.)

Fun working with YOU, though, snookums.

Hitch

You could have used combining circumflex and kept praying ;)

B̂

It looks like crap on this site in my browser with whatever sans-serif font it's using, but pretty decent on a separate HTML file with the default (Deja Vu Serif) font:

Tried it. Doesn't work for eBooks, at all. And the customer was NOT interested in hedging it that way, without the ability to place the circumflex directly OVER the b. (sigh).

Hitch

There is a way to do it. Modify a font and embed it.

Jon, do you also try to teach your grandmother to suck eggs? Of course I could do that--right up until that hits the KF7 wall, or the device-that-doesn't-display-embedded-fonts wall.

The customer doesn't want that, and neither do I. Even Cambria's math glyphs don't have the bloody thing. There is no b-hat unicode and believe me, we looked.

Hell, there are Stack Exchange whinges about it!

Hitch

OK.... I am definitely NOT making any kind of implication as to your actual age or experience.... but.... was "sucking eggs" a real thing?!?!? I always thought it was just a saying: "Go suck eggs!" :blink:

It MIGHT just be saying, because it makes no sense. It's too difficult. It's simpler to break the egg and beat it if you want it raw, but scrambled egg is edible with no teeth and as old as pita (flat bread), i.e. ancient.
https://en.wikipedia.org/wiki/Teachi...r_to_suck_eggs
Pizza pre-dates tomato. It's a corruption of pita, which originally was just flat bread, not "pocket bread".

Would an SVG graphic of the B-hat work on a Kindle in KF8 and KFX?

And as for Amazon, they really Fed up with embedded fonts. They should automatically select the publisher font option.

I agree vis: the Publisher Font option and it's....infuriating.

No, you can't use inline SVGs, at least, not as of my last test. Everytime you put a bloody SVG (which is, mind you, allegedly "supported" for KF8/KFX!), you get an inadvertent page-break (screen break). Works fine for full-page/screen images, natch--but it's utterly worthless for inline.

Trust me, Jon--I swear to you, this is something we've searched high and low on. Unless KDP has changed something around SVG, in the last...say, 4 months, the ONLY solution so far is (regular, JPEG/GIF/PNG) images and yes, I hate it.

Hitch

That is too bad. That makes some eBooks not a good as the could/should be. It can be difficult to read a graphic that's meant to be regular text. This is one reason I stick with ePub. I forget which book it was,but there was a phrase in Hebrew and the graphic used was not good on screen. I removed the graphic and used an embedded font. It worked well in ePub.

Sorry Amazon makes these sorts of things a pain in the ass.