foreign characters not showing up?

[sovre]

I have some epub formatted books with foreign characters. The texts look fine in Calibre, and also if read on my iPod Touch, but on my Sony 950 all of the foreign characters show up as question marks, which becomes quite distracting when I am trying to read. I'm trying to figure out if there is a way to fix this? Thanks in advance to anyone who can share the solution.

[Toxaris]

The Sony should not have any issues with foreign characters. Are you sure that the encoding is UTF-8?

[sovre]

I don't know, as I received the file already formatted. What tool would I use to verify this? Is there a way for me to convert it to Unicode 8 if necessary?

[roger64]

Originally Posted by sovre

I don't know, as I received the file already formatted. What tool would I use to verify this?

On Linux, you can use the command tool file which is intended to determine file type (see man file)
Set yourself on the path (using cd command), then use the following command:

file name_of_the_file

http://linux.die.net/man/1/file

There is one online converter using this command
http://www.cometdocs.com/file.htm

[charleski]

The most likely answer is that your ePub doesn't have an embedded font. ADE's default glyph set is very narrow and only really suitable for books in English.

If you can edit the ePub, then you can embed a font yourself. Or alternatively you could load kartu's PRS+ firmware into your 950 (see the Sony Dev sub-forum on this board), which allows you to upload a default font of your choice to the reader.

[Toxaris]

Sorry, I don't agree. I have character test epub which contains all characters used in west-european languages. All the characters display on my old PRS-300 without a problem. I haven't tried Eastern-European or Cyrrilic, so that could be the case.

For me embedding a font would be the last option, due to the extra size and possible copyright issues.

Sovre, what characters are you trying to display? Perhaps then we can identify the problem better.

[charleski]

The default font covers enough of Latin Supplement 1 to get by in many Western European languages, but there are gaps and it has no support for Latin Extended-A.

[sovre]

I am trying to read an English translation of a Pali text. The proper names in Sanskrit which have diacritical marks are the ones which cannot be displayed.

I have PRS+ loaded as the firmware for my reader, but I do not know how to upload a new font?

If you could explain how to do this or send me a link to instructions, I'd appreciate it!

I notice in the PRS+ settings there is something called "User EPUB style (CSS file)" under Book Viewer Settings, but I'm not sure what this means or what its intended use is.

[charleski]

Download the files I've attached. Hook your reader up to the computer via USB and open the READER drive. Make a new directory called fonts in the top level of that drive, then open the CharisSIL zip file and copy the four .ttf files it contains into that directory. Go back to the top level of the READER drive, and go down to database/system/PRSPlus/epub . Copy the CharisSIL.css file into that directory. Close the Sony Reader app if it auto-launched, and then eject the READER drive from the computer and disconnect it when it says it's safe to do so.

If you now go to User EPUB Style you'll have an option for CharisSIL. Select that and the font will be applied to every book you open that doesn't have a font embedded itself. If you already have the book open that you want to read, open a new one, then go back to the home screen and open the book you're reading.

[sovre]

Hi Charleski,

Thanks for taking the time to give me these clear instructions. I followed all you said, and now have that font selected as my epub style.

Only one problem: I am still getting a "?" for the letters which have diacritical marks in that text.

Oddly enough, I did notice this: the font has changed, but it changed in texts which were fine and not giving me a problem. And the texts which use the new font do not appear to be as readable as those using the old font. The paragraphs look somehow "clumped together," instead of there being a bit of breathing space between the lines as before.

[sovre]

While reading another thread on another subject I got an idea about this problem.

And then I tried this:

I copied the epub file and pasted it into MS Word, and then saved it as an rtf file. I then converted the rtf file to epub using calibre and sent it once again to my Sony.

Now all words show up fine, with proper accenting. No more question marks.

But the solution is not ideal because some of the formatting was lost, and I have footnote numbers showing up in odd places and causing paragraph breaks where they should be none.

[charleski]

Originally Posted by sovre

Only one problem: I am still getting a "?" for the letters which have diacritical marks in that text.

Originally Posted by sovre

I copied the epub file and pasted it into MS Word, and then saved it as an rtf file. I then converted the rtf file to epub using calibre and sent it once again to my Sony.

Now all words show up fine, with proper accenting. No more question marks.

This suggests that the problem actually lies in the encoding, as others mentioned earlier in the thread. As you've discovered, calibre's automatic conversion can easily break if there's any problem in the source. Unzip the epub and extract one of the html files, then open that in Notepad++. Can you still see the proper diacritics in the text? Clicking on the Encoding menu, does it say 'Encode in UTF-8 without BOM'? What is the first line of the file? It should be something like
<?xml version="1.0" encoding="utf-8" standalone="no"?>

Originally Posted by sovre

Oddly enough, I did notice this: the font has changed, but it changed in texts which were fine and not giving me a problem. And the texts which use the new font do not appear to be as readable as those using the old font. The paragraphs look somehow "clumped together," instead of there being a bit of breathing space between the lines as before.

Yeah, this is a problem with the font. Charis supports an extremely wide range of glyphs, has bold and italic variants, and is free, so I could post it here. But it was designed to be compact, and therefore doesn't have its height metrics set properly (they've released an even more compact version, which is worse). If you look around at the fonts you have installed on your system you may well find one that will work better, just copy those over (you'll need normal, italic, bold, and bold-italic variants, which will be different files) and edit the file names in the css file.

[sovre]

Ok I downloaded Notepad ++.

Yes it says Encode in UTF-8 without Bom under "encoding"

The first line is: <?xml version='1.0' encoding='utf-8'?>

But I don't know how to access the text itself using this text editor. I only seem to be able to see information about the encoding and fonts. Opening the text in another text viewer I have which displays the entire text, I can see the diacritical marks fine.

Wouldn't this indicate the encoding of the text is ok? Or is there a problem with it?


Also, are there any Unicode 8 fonts you recommend as being clear and readable with the Sony Reader? And is there some kind of basic template I can use for creating a css file to go with the font (I've never done this before and don't know how!).

Thanks.

[Toxaris]

Can you give an example of the characters you want to display that give a question mark? Give a few examples, if you like. Some simple, some exotic (your interpretation of course). Perhaps we can give a better advise then. I could add it to my character test epub and see what the result would be on my Sony.

[sovre]

I've uploaded the file--I think that will make it easier for you to diagnose the problem, because you can see how it displays on your own Reader.