What the estore does to your epub

[Difflugia]

In response to some discussion in this thread, I downloaded the same free epub from as many ebook sellers as possible and looked at the differences inside.

I downloaded A Perilous Proposal by Michael Phillips from Barnes & Noble, eBooks.com,Kobo (both ADE and kepub), Christianbook, Google Play and Amazon.

The stores that I'm aware of but didn't try are:

• iTunes: I don't have iTunes installed and if I did, I can't remove iTunes DRM.
• Books-A-Million doesn't offer any free ebooks as far as I can tell.
• Lifeway.com has its own reader with DRM that I can't remove.

I downloaded all of the books into Calibre using "The Tools" and then unzipped each epub to compare contents. My comparison tool of choice is Beyond Compare, which will compare entire directories and even images.

The ADE-protected epubs downloaded from Kobo and Christianbook.com were nearly identical and so I'm assuming that they most closely match the epub uploaded by the publisher. The only differences I could find were Adept resource strings from the ADE system. My caveat about Kobo is that the ADE book is occasionally the kepub version described below and there's no way to tell beforehand if that's what you'll get.

The copy from eBooks.com had slightly different metadata (the title was "Perilous Proposal, A" and the author was "Phillips, Michael"). The files had DOS-style line endings (CR-LF) while the others were UNIX-style (LF). Other than that, this copy was the same as the previous two. As an extra bonus, if the publisher offers a PDF, then it's usually included as a separate download. Unfortunately, most publishers don't sell PDFs anymore, so fewer and fewer books have them available (this one didn't).

The epub from Barnes & Noble makes minor changes to the stylesheet, changing "text-align:left" to "text-align:justify" and changing some margins. B&N also apparently does some filtering on the character sets used in the book. Em dashes ("—") are changed to the HTML code "—".

The copy from Google Play had extra anchors in the XHTML, presumably for the Google Books software. Google also did some similar filtering to B&N, but Google's is broken and the em dashes are just gone. Here is the same paragraph, first from the Kobo book:

<p>“That was a small detail of rebs. They’d have killed you sure if I hadn’t come along. But they’re gone now. You don’t have to worry about them no more. I was out ahead of our company. That’s what I do—I’m a scout. And I tend the horses. I was scouting when I ran into them. Lucky for you I did too. In case you hadn’t noticed, they were wearing the <a id="page_60"/>grey of the Confederate rebels. We’re wearing the blue of the U.S. infantry. So are you.”</p>

now from Google Play:

<p>“That was a small detail of rebs. They’d have<a id="GBS.0049.02"/> killed you sure if I hadn’t come along. But they’re gone now. You don’t have to worry about them no more. I was out ahead of our company. That’s what I doI’m a scout. And I tend the horses. I was scouting when I ran into them. Lucky for you I did too. In case you hadn’t noticed, they were wearing the <a id="page_60"/>grey of the Confederate rebels. We’re wearing the blue of the U.S. infantry. So are you.”</p>

It's also clear that Google re-encodes the images with more JPEG compression than the original files. It's probably not a big deal, but JPEG uses lossy compression, so the Google images may be slightly degraded. The file sizes on the original images:

89709 Titlepage.jpg
9843 common.jpg
9264 common1.jpg
213011 cover.jpg

and the Google epub:

67503 Titlepage.jpg
1558 common.jpg
1147 common1.jpg
119889 cover.jpg


An extra bonus with Google Play is that some books (including this one) include a PDF. This is tempered (for me, anyway) by Google's conversion of all PDFs into page images, even if the PDF started out having been typeset (like this book obviously was).

The prize pig is the kepub. Here's the same paragraph we looked at before:

<p xmlns="http://www.w3.org/1999/xhtml"><span id="kobo.40.1">“That was a small detail of rebs. </span><span id="kobo.40.2">They’d have killed you sure if I hadn’t come along. </span><span id="kobo.40.3">But they’re gone now. </span><span id="kobo.40.4">You don’t have to worry about them no more. </span><span id="kobo.40.5">I was out ahead of our company. </span><span id="kobo.40.6">That’s what I do—I’m a scout. </span><span id="kobo.40.7">And I tend the horses. </span><span id="kobo.40.8">I was scouting when I ran into them. </span><span id="kobo.40.9">Lucky for you I did too. </span><span id="kobo.40.10">In case you hadn’t noticed, they were wearing the </span><a id="page_60"></a><span id="kobo.41.1">grey of the Confederate rebels. </span><span id="kobo.41.2">We’re wearing the blue of the U.S. </span><span id="kobo.41.3">infantry. </span><span id="kobo.41.4">So are you.”</span></p>

There's a Calibre plugin that does a good job of cleaning that stuff up into a regular epub, or for those of opposite bent, there's one to add it to regular epubs and make kepubs.

Finally, there's Amazon. While the Kindle book starts life as an AZW3 file, the Kindle Unpack plugin will retrieve an epub that's very close to the source that was used to create it. If the original was a valid epub (which if the book is from a major publisher, it usually is), then you'll get a valid epub out. The only difference I noticed in the text is an em dash filter again (which works). Additionally, the images are compressed like Google's and the internal file names aren't preserved in the epub->Kindle conversion. For you folks that buy Kindle books and convert them to epub with Calibre, you may want to try Kindle Unpack first to see what you get.

[eschwartz]

Well, the JPEG compression saves me having to do it...

[Notjohn]

That's a wonderful comparison. Thank you!

Interesting too that most (all?) New York publishers upload epubs to the KDP.

[eschwartz]

They upload EPUB through FTP, actually. They have special permissions.

[Difflugia]

Originally Posted by Notjohn

Interesting too that most (all?) New York publishers upload epubs to the KDP.

If they have it for sale in other stores, they have to make an epub anyway. Since Amazon's conversion works well with them, I assume there's just no reason to make a separate Kindle version.

[Jane12]

Originally Posted by Difflugia

I downloaded the same free epub from as many ebook sellers as possible and looked at the differences inside.

Thanks so much for doing this!

[Hitch]

Originally Posted by eschwartz

They upload EPUB through FTP, actually. They have special permissions.

Actually...

Most of them are FTP'ing, simultaneously, through distributors, to multiple retail outlets. Hence, ePUBs rather than MOBI.

ePUB translate fine to MOBI if what you're doing doesn't need "anything." If you need to make a "One Ring to Rule them All" ePUB, then you have an issue, as you will run into problems with media-queries in the same ePUB, if it has to pass ePUBcheck. This means--by definition--that the ePUBs being uploaded are, of necessity, more vanilla than those which are specifically made for MOBI. (Hence: non-fiction-type books which are lower in standard than need be, in order to fit into the ePUB FTP mold.) Alternatively, the retailer has to make two ePUBS w/two ISBNs; one expressly for mobi, replete with media-queries, and one without, for the ePUB vendors.

Given what we've all seen, from BPH's, in terms of formatting, this hardly constitutes an imprimatur of legitimacy. More an imprimatur of laziness and an utter failure to proof the books.

Hitch

[eschwartz]

Why are two ISBNs needed, if it is the exact same book? As well say that the MOBI needs a separate ISBN from the AZW3?

[Hitch]

Originally Posted by eschwartz

Why are two ISBNs needed, if it is the exact same book? As well say that the MOBI needs a separate ISBN from the AZW3?

Because otherwise, eschwartz, Amazon will ingest the not-meant-for MOBI ePUB and convert it, and then have two versions of the same book on the shelves. My point was, if you (the publisher) need a "real" ePUB that's going to be used for B&N, iBooks, etc., and you are going to be FTP'ing to your distributor, all at once, you end up with a sub-optimal MOBI, because you cannot use media-queries to OPTIMIZE the mobi file. You can only do that if you provide Amazon with an ePUB for it, alone (using the correct ISBN), so that only ONE of the ePUBs gets converted.

Having used the FTP ingestion method, I can tell you that it's not meant for "exceptions." For example, half the distributors out there can't/won't intake MOBI, at ALL--they'll only intake ePUB, which means that an author can't do anything beyond vanilla. No FF, no comics, nada. The only way around this is to use an ePUB ISBN, a MOBI ISBN, and ensure that the MOBI ePUB, w/media queries, gets ingested at Amazon for conversion.

An ePUB with media queries is NOT the same book as the ePUB without them. That's the point. Whereas a MOBI and an AZW are the same book, simply modified, from the delivery point (Amazon), not the ingestion point, to target the receiving device. Not using media-queries to target the OUTPUT of the conversion.

See what I mean?

Hitch

[eschwartz]

Yeah, I see that people need to get their act together regarding exceptions... they *should* be able to choose which EPUB goes where, although I suppose that would require a perfect world.

I also wonder how difficult it would be to process a media-queried EPUB back into the non-media-queried version, possibly dynamically in the upload.

Maybe I am overthinking this.

[Hitch]

Originally Posted by eschwartz

Yeah, I see that people need to get their act together regarding exceptions... they *should* be able to choose which EPUB goes where, although I suppose that would require a perfect world.

I also wonder how difficult it would be to process a media-queried EPUB back into the non-media-queried version, possibly dynamically in the upload.

Maybe I am overthinking this.

My experience with trying to somehow automagically make a media-queried ePUB into a not-media queried ePUB has basically led me to...

making a Kindlegenned mobi. ;-)

Hitch

[JSWolf]

I have seen a lot of ePub with empty media queries. But then, the CSS is probably a robo CSS that's the publisher's catch all CSS.

[photon_man62]

This is a slightly old topic, but it's one of the top Google results so I'd like to share my info here. Amazon AZW3/KF8 files are much smaller than EPUB/classic MOBI due to their recompression of images, as noted in the OP.

Here's some examples I pulled from a cookbook. Retail EPUB (62MB) vs. Retail AZW3 (14MB)
http://screenshotcomparison.com/comparison/159478

It's not bad at all for the space it saves, but it's worth noting if you're releasing something photography-related.

[Hitch]

Originally Posted by photon_man62

This is a slightly old topic, but it's one of the top Google results so I'd like to share my info here. Amazon AZW3/KF8 files are much smaller than EPUB/classic MOBI due to their recompression of images, as noted in the OP.

Here's some examples I pulled from a cookbook. Retail EPUB (62MB) vs. Retail AZW3 (14MB)
http://screenshotcomparison.com/comparison/159478

It's not bad at all for the space it saves, but it's worth noting if you're releasing something photography-related.

Sorry: why do you think that the KF8 file is smaller than the "classic MOBI" file?

Hitch

[photon_man62]

Originally Posted by Hitch

Sorry: why do you think that the KF8 file is smaller than the "classic MOBI" file?

Hitch

Amazon adds extra JPG compression to images in KF8 files. So in this example, a cookbook which used to take up 62MB (because of all the photos of ingredients, recipes etc.) in the "classic MOBI" format now takes up only 14MB as a KF8 file. The same cookbook still takes up 62MB in other ebook stores that use EPUB.

In the link I posted, I gave an example of the same pictures from an EPUB (mouse out) and from a KF8 file (mouse over the photo). The compression isn't very bad so you normally don't see a big difference in image quality.

However when you look closely, you will see that there is a difference. Hold down Ctrl and press + a few times to zoom in. It's easiest to notice in picture #2 if you look at the background. When you mouse over the image you will immediately see that there is a completely different background texture. Look, this is the top-left corner of picture #2:



The difference in picture quality isn't really a problem during normal viewing, especially on a black&white e-ink reader. However if the ebook you're publishing is very reliant on pictures, you should still keep this in mind.