Character Encoding: How to fix it?

[Genre fan]

I have Sony PRS-950 and PRS-350 devices.

In the last year, I've been getting books with odd characters instead of punctuation, which make the books/chapters difficult to read. In playing around with my browsers and View -> Encoding menus, I have figured out that it has something to do with the Character Encoding within the epub files.

Example: I’ve is printed instead of I've.

’ for apostrophe
“ the opening of a quotation,
� for closing the quotation,
and I think — is for a hyphen.

When a sentence had “’m for " 'm at the beginning of a speech (when the character was slurring his words) it took me a while to figure out how it was supposed to read.

This was in one recent book:

“’Sides, ’tis only for a moon. That ain’t long.�

Translation: " 'Sides, 'tis only for a moon. That ain't long."
See what I mean about it being really hard to read?


I buy books from several ebook stores and I borrow from the library.
The problem may be the entire book, but it is usually restricted to a few chapters, with rare occasion where the encoding changes within a chapter. Usually it is for a whole chapter, not part, and it can be seen in chapters not consecutive to each other.

It occurs whether the book is downloaded directly to my 950 reader or if I load it to either reader from my computer(s), which are all Mac OS X of several versions from 10.4 to Mountain Lion.
Since it happens when the book is downloaded directly, I figure the operating system of my computer is not relevant.

There are several publishers involved, though http://www.baenebooks.com/ (no DRM!) has not so far been one of them, IIRC. I haven't actually purchased Baen ebooks from any source except the publisher, so there is a slight possibility that the problem is dependent on the store/source and not the epub file as originally published. I know I get this in books from the Sony Reader store and from Kobo Books. I haven't purchased enough books from other vendors in the last few months to have a large enough sample to say anything about other stores.

However, if I view the books with any viewer on my computer, the encoding is the same. I've read them in Calibre (after stripping DRM -- for my personal use only! -- so that I can actually look at the books in the viewer), in the Sony Reader App, and in Adobe Digital Editions 2.0. It's always the same.

I believe the encoding is inherent to the files. I would like to fix this if I can to make the books I've purchased more enjoyable to read on my ereaders.

Any ideas?

BTW, to paraphrase Bones McCoy, "I'm a doctor, not a software engineer!". It would be really helpful if any suggestions don't assume that I know what you are talking about. Links or specific steps would be very helpful. I have Calibre on my computer, but I am very much a beginner to using it. I can add books and change some of the obvious metadata, but that's about it. I looked a bit at the online user manual and it has stuff about converting books, but it's not clear to me if that's what I should try to do.

[Dngrsone]

It's a problem that plagues all of us at one point or another.

Fixing it can be difficult, depending on formats and tools available, and time-consuming.

I often use Calibre to convert books from one format to another, with decidedly mixed results.

If I can get away with it, I prefer my book in rich text format (.rtf), which I can then open with a simple text editor like gedit (I use it in Linux; it is a bit more powerful than Microsoft's Notepad) and do search and replace routines.

Calibre does have some of this functionality built-in, I believe, but frankly I use it only as a conversion tool and am not versed with all of its functionality.

[Doitsu]

Originally Posted by Genre fan

BTW, to paraphrase Bones McCoy, "I'm a doctor, not a software engineer!". It would be really helpful if any suggestions don't assume that I know what you are talking about. Links or specific steps would be very helpful. I have Calibre on my computer, but I am very much a beginner to using it. I can add books and change some of the obvious metadata, but that's about it. I looked a bit at the online user manual and it has stuff about converting books, but it's not clear to me if that's what I should try to do.

You could try the following, download Sigil, open one of the DRM-free books with encoding problems with it and click the large green check mark in the Sigil toolbar or press F7 to check for errors. If Sigil doesn't find any errors and all typographical quotes are being displayed correctly in Book View mode, press F8 and add and delete a space somewhere to trigger the change indicator and save the .ePub file. (This should fix any encoding issues.)

Then transfer the file to your Sony to see if the text is being displayed correctly.

[theducks]

Your problem was that the codepage was not declared (or done incorrectly).
Calibre has a conversion setting (Look and Feel) where YOU can declare which to use when converting the book .

I suggest using this setting from the individual book conversion screen (defaults override) and NOT set it in Preferences (defaults)

CP1252 is a good first shot at this problem

[eschwartz]

You can also edit the book directly from calibre, as of version 1.15, using the new Edit Book tool (shortcut key is "T") -- no need to install Sigil. It includes a Check Book tool to find problems, and a preview. The Check Book tool gives you the option to fix structural problems automatically, and one of the things it does is fix the encoding.

[dreamcast]

It's been about 8 years since this thread started, but I wanted to say a huge thank you to everyone involved as it helped me resolve the same issue on a file that I was struggling with. I used Doitsu's method and the book is finally readable :')

[jharris]

Thanks for helping me solve this encoding problem - when I email epub files to my Kindle Paperwhite 4 (10th generation) I would see special corruption-like characters appear in the text. Apparently epub file encoding is very sensitive - at least on the Kindle readers. Using Calibre's "edit," and "check" feature fixed it. Also did a Calibre convert to epub just for fun.
Sigil also has a check epub/code feature that would probably help.

Very helpful tip, thanks. JH