Special characters and conversion

[MGA]

Dear friends,

I have a problem with converting a book (from docx to epub)

The book in question contains some words in the ancient Indian language Pali, which have a few special characters that are difficult to get properly displayed. Some examples of the characters in question are these: ṅ ṭ ṃ ṁ ṇ

When I view the epub in my e-reader (Kobo Clara 2E) each of these characters are simpy ignored and words are "merged together". For example: the word tomato would turn into toato. Strangely enough, when I open the file on the Calibre ebook reader, it works fine. It is only when I use the Kobo that it malfunctions.

I have some other books on the Kobo, where these characters display very fine, so I know that it is not beyond the capabilities of the device in itself.

Would be very grateful for any advice on this!

[MGA]

In relation to this statement from the original post:

For example: the word tomato would turn into toato.

But I also spotted that this letter ṭ can turn into ñ instead of disappearing, but not consistently so.

[DNSB]

Sounds as if you are trying to display glyphs that are not supported by the current font used on your Kobo ereader.

[kovidgoyal]

You need to embed a font in the book that has those characters. Either embed it in the word document itself and conversion should preserve it or embed after conversion using the editor and apply the right font styles, or use the conversion option to embed all referenced fonts, provided you have the font on your system somewhere and the docx file actually references the correct font by name but just doesnt embed it.

[theducks]

Those seem to be Language relevant.
Conversions from earlier TXT files sometimes needed you to set the CHARSET at the beginning of the Add. What Language was the DOCX using?

I was guessing ligatures (ffi,ll...) but I don't know of one that includes an M, so that rules out that reason for 'missing'

[Julie Paradise]

This is a two-step problem: First you need to make sure that the correct Unicode character is used in the source/text itself (so not DIY combination of letter-x+point-below), and then you need to make sure that the system/platform/device that displays this text and its diacritical special glyphs also has a font that contains this Unicode glyph.

[MGA]

Originally Posted by Julie Paradise

This is a two-step problem: First you need to make sure that the correct Unicode character is used in the source/text itself (so not DIY combination of letter-x+point-below), and then you need to make sure that the system/platform/device that displays this text and its diacritical special glyphs also has a font that contains this Unicode glyph.

Thank you so much for taking the time and making the effort. As I understand it, the first part of the problem, regarding whether the correct characters are used in the original input file: Yes, they are, and they are displayed very well there, using the ordinary Arial font.

And as for the second side of the problem: What would then happen if I specifically downloaded a font and added it to my own device, and then share the file in question with someone else? Would they end up with the same error, or is it somehow possible to "attach" a font with the file?

Sorry if my questions are on a very elementary level.

[theducks]

Originally Posted by MGA

Thank you so much for taking the time and making the effort. As I understand it, the first part of the problem, regarding whether the correct characters are used in the original input file: Yes, they are, and they are displayed very well there, using the ordinary Arial font.

And as for the second side of the problem: What would then happen if I specifically downloaded a font and added it to my own device, and then share the file in question with someone else? Would they end up with the same error, or is it somehow possible to "attach" a font with the file?

Sorry if my questions are on a very elementary level.

The proper way to avoid the issue is to EMBED the font into the book.
Depending on the font face, get out your wallet if you want to distribute the book. Font license can get expensive. Most fonts are copyrighted. and it geta worse. The license might needed for up to 4 of them (normal, bold, italic bold-italic)

[Julie Paradise]

Originally Posted by theducks

The proper way to avoid the issue is to EMBED the font into the book.
Depending on the font face, get out your wallet if you want to distribute the book. Font license can get expensive. Most fonts are copyrighted. and it geta worse. The license might needed for up to 4 of them (normal, bold, italic bold-italic)

Yes, either embed the font (or at least glyphs used) or share it as PDF, depending on the circumstances.

Regarding the licence: Yes, licensing is another thing to consider. Thus the safest way would be to not try and use a fancy font with a licence, but just take one of the few very known fonts that basically have anything and everything covered of the Unicode chart. That leaves you with (admittedly not so pretty) fonts like Times New Roman, Arial, Noto, etc.

[MGA]

Originally Posted by theducks

Those seem to be Language relevant.
Conversions from earlier TXT files sometimes needed you to set the CHARSET at the beginning of the Add. What Language was the DOCX using?

I was guessing ligatures (ffi,ll...) but I don't know of one that includes an M, so that rules out that reason for 'missing'

Thanks for taking the time and making the effort!

I am quite a novice in this, but I managed to access the "document.xml", and there I found this information:

encoding="UTF-8"

It seems to me that this might be very relevant...?

[MGA]

Originally Posted by theducks

The proper way to avoid the issue is to EMBED the font into the book. (...)

Excellent, I will try to examine that option.

[MGA]

Originally Posted by Julie Paradise

Yes, either embed the font (or at least glyphs used)

Exccellent. I hope this is not too complicated. But will examine it. And from what I read previously, it would perhaps be best to do this in the DOCX, before converting it to EPUB?

Regarding the licence: Yes, licensing is another thing to consider. Thus the safest way would be to not try and use a fancy font with a licence, but just take one of the few very known fonts that basically have anything and everything covered of the Unicode chart. That leaves you with (admittedly not so pretty) fonts like Times New Roman, Arial, Noto, etc.

That is completely fine, thanks for explaining.

[Quoth]

No need to embed in the docx, which makes opening times terrible.
As long as the same fonts are on the PC running Calibre (always true if the same computer), it will find and embed the fonts if asked.

Obviously if for publication you need either free fonts or a licence that covers ebook distribution (which might be incompatible with Kindle Publishing if the fonts have to be obfuscated or encrypted).
It doesn't matter for personal use.

[DNSB]

It would be easier to embed the font in your ePub after the conversion. You can use the calibre ebook-editor to do this. As @Quoth mentioned, if this is not for your personal use, you would have to check the licensing on the font.

[theducks]

Polish can do it also