Amazon's flawed conversion of epubs

[A Lurker]

I purchased an ebook from Kobo (epub format), imported it into Calibre (using a plugin to remove the DRM), then sideloaded it to my Kindle using my Send To Kindle email address.

The book displayed oddly on my Kindle. All of the apostrophes and quotation marks were replaced by a series of symbols: ’ (a-circumflex, Euro, superscript trademark). For example, the sentence: "I'm not sure why you don't run away." displayed as: ’I’m not sure why you don’t run away.’

I took the same file I'd emailed my Kindle and opened it with Calibre's native viewer. The punctuation displayed correctly there. That suggested the original file is fine; the problem occurs when Amazon converts this epub to awz3.

As a test, I created a mobi version with Calibre and sent it to my Kindle using the desktop Send To Kindle app. (Send To Kindle email no longer allows uploads of mobi files, but the desktop app does.) This time the book displayed properly, confirming the error was an artefact of Amazon's converting the epub.

Bottom line, I found a solution, so I'm not posting here to ask for help. I just want to share my experience in case anyone else runs into this issue. If Amazon screws up your epub, trying sending a mobi version with the desktop app (as long as that remains as possiblity... I believe the desktop app will also eventually drop mobi support).

Out of curiosity, has anyone else come across this? I've had a book's formatting display differently on a Kindle compared to its appearance in Calibre, but this is the first time I've had actual text get screwed up.

[DNSB]

I thought this topic had been beaten to death with multiple threads about the topic. The most likely problem is that your epubs are not marked as using UTF-8. You can use calibre's Modify Epub plugin to correct this before sending the epubs to Amazon (turn off all the options except Encode HTML in UTF-8). BTW, the epub standard requires an epub to use either UTF-8 or UTF-16 encoding though I've only seen 2 epubs that used UTF-16.

One other item is that calibre's ebook-viewer is chromium based and like most web browsers tends to hide many errors. It is not a good test bed for how an epub will display on an eInk reader or most apps.

[tekkasit]

In short, your source ePub file is not in a strictly well-formed format.

Amazon's Send to Kindle service supports EPUB files; however, for historical reasons, it still assumes ISO-8859-1 encoding if no encoding is specified, while most "assume" as UTF-8 and continue. This creates malformed formatting errors for special characters, especially "smart" punctuation.

[haertig]

Originally Posted by DNSB

One other item is that calibre's ebook-viewer is chromium based and like most web browsers tends to hide many errors. It is not a good test bed for how an epub will display on an eInk reader or most apps.

Is there a good test bed to check for errors, other than putting the book on an eReader and doing manual observation? Something that can be run on a desktop computer?

[innocenat]

Originally Posted by haertig

Is there a good test bed to check for errors, other than putting the book on an eReader and doing manual observation? Something that can be run on a desktop computer?

Download Kindle Previewer 3 and open your EPUB with it.

Note that Kindle Previewer 3 is more strict than the current Amazon processing pipeline, files that failed to convert with Kindle Previewer may be converted with Amazon's Send to Kindle. But it is a good place to start nonetheless.

[JSWolf]

Originally Posted by innocenat

Download Kindle Previewer 3 and open your EPUB with it.

Note that Kindle Previewer 3 is more strict than the current Amazon processing pipeline, files that failed to convert with Kindle Previewer may be converted with Amazon's Send to Kindle. But it is a good place to start nonetheless.

Kindle Previewer 3 does not always work well. It's more forgiving then an actual Kindle. The solution is to use Modify ePub to make sure the ePub is set to UTF-8 and then you can send your ePub to Amazon.

[innocenat]

Originally Posted by JSWolf

Kindle Previewer 3 does not always work well. It's more forgiving then an actual Kindle. The solution is to use Modify ePub to make sure the ePub is set to UTF-8 and then you can send your ePub to Amazon.

It works very well as a testbed to check for errors, which was the question I answered.

Also can you give example on where it is more forgiving, in relation to EPUB conversion error? Last I checked Kindle Previewer always error on out of order TOC, while it will now convert with Send to Kindle (with TOC stripped).

[MSWallack]

Query: So let's assume that I've acquired an EPUB formatted book that displays properly in Calibre's viewer but not on the Kindle (because of the oddly formatted special characters). What is the correct workflow to be sure that the EPUB is encoded as UTF-8 (and converting it if necessary) before sending it to the Kindle?

[innocenat]

If you are already using calibre then using the Modify EPUB plugin to add UTF-8.shoukd fix the problem.

If you are not using calibre then using https://kindle-epub-fix.netlify.app/ is probably easier. (Disclaimer: I built that).

[JSWolf]

Originally Posted by innocenat

It works very well as a testbed to check for errors, which was the question I answered.

Also can you give example on where it is more forgiving, in relation to EPUB conversion error? Last I checked Kindle Previewer always error on out of order TOC, while it will now convert with Send to Kindle (with TOC stripped).

No it doesn't always work well. It gets line height wrong as line height works in Kindle Preview but not on a Kindle. There's probably other things that work differenly with Kindle Previewer vs a Kindle.

[innocenat]

Originally Posted by JSWolf

No it doesn't always work well. It gets line height wrong as line height works in Kindle Preview but not on a Kindle. There's probably other things that work differenly with Kindle Previewer vs a Kindle.

You are entirely missing the point. The topic is about mis-encoded characters (and maybe other Send to Kindle errors), which Kindle Previewer is perfectly capable of checking.

[JSWolf]

Originally Posted by innocenat

You are entirely missing the point. The topic is about mis-encoded characters (and maybe other Send to Kindle errors), which Kindle Previewer is perfectly capable of checking.

Why bother? It's a waste of time. It's just easier to use Modify ePub to make sure the ePubs are set to UTF-8 and then send them to Amazon to convert.

[A Lurker]

Originally Posted by DNSB

You can use calibre's Modify Epub plugin to correct this before sending the epubs to Amazon (turn off all the options except Encode HTML in UTF-8).

Originally Posted by innocenat

If you are already using calibre then using the Modify EPUB plugin to add UTF-8.shoukd fix the problem.

Where do I get the Modify Epub plugin?

Thanks.

[JSWolf]

Originally Posted by A Lurker

Where do I get the Modify Epub plugin?

Thanks.

Within Calibre. Preferences > Plugins > Get new plugins Search for Modify ePub and you'll find it and can then install it.

[DNSB]

Or if you are enthused, you can add the EpubCheck plugin to Sigil or calibre's editor. For most places, if it doesn't pass epubcheck, they won't even try to publish it. If you don't want to use Sigil or calibre, you can run epubcheck from a command line though I generally reserve that for pre-processing a mass of epubs.

For the last few years it's been better but the memory of how many "professionally" produced epubs failed epubcheck, often in a rather spectacular display of error messages still drives my lack of enthusiasm for most publishing houses ability to produce a half decently formatted ebook.