Does Kindle actually use epub format?

[Notjohn]

Just for fun, I used Kindleunpack to deconstruct a book of mine, converted on the KDP platform for sale on Amazon. I upload an epub to the KDP, since I need the epub for the other retailers, and since it happens to be the easiest way to upload an entire book in a single package, rather than as a zip file.

Kindleunpack produced two files, one in a folder called Mobi7, the other in a folder called Mobi8 which I take to be the equivalent of KF8 format (for the Fire and other newer devices and apps). To my astonishment, the file in the Mobi8 folder had the extension *.epub.

Is Amazon actually delivering an epub to customers who are reading on the Fire and related devices/apps?

(The Mobi7 folder contained a much smaller file with the extension *.htm, plus an OPF and an NCX file.)

[Nate the great]

The latest versions of Kindle gen will make a file with up to 3 ebooks inside:

Mobi7, KF8, and Epub (if that was the original source)

I think you got a file that only has Epub and Mobi7. That is unusual but not completely surprising.

[DaleDe]

Originally Posted by Notjohn

Just for fun, I used Kindleunpack to deconstruct a book of mine, converted on the KDP platform for sale on Amazon. I upload an epub to the KDP, since I need the epub for the other retailers, and since it happens to be the easiest way to upload an entire book in a single package, rather than as a zip file.

Kindleunpack produced two files, one in a folder called Mobi7, the other in a folder called Mobi8 which I take to be the equivalent of KF8 format (for the Fire and other newer devices and apps). To my astonishment, the file in the Mobi8 folder had the extension *.epub.

Is Amazon actually delivering an epub to customers who are reading on the Fire and related devices/apps?

(The Mobi7 folder contained a much smaller file with the extension *.htm, plus an OPF and an NCX file.)

You can read about MOBI, KF8 in our wiki. A file built by KindleGen includes 3 formats. The mobi7, KF8, and the original submitted ePub. KindleStrip will remove the source ePub to reduce the size and KindleUnpack will split the 3 parts into separate files and if desired it can further break them down in their source equivalent parts.

Amazon does not deliver what you make with KindleGen but may deliver either one of the other two or both of them in a file depending on what device or devices you own.

Amazon KF8 is really a ePub content inside a mobi wrapper (container) which is explained in the wiki. Amazon never delivers the epub for use by the end user.

Dale

[Notjohn]

As noted, Kindleunpack shows me only two folders, Mobi 7 and Mobi 8, and the latter includes only a file with the extension *.pub.

Yes, I uploaded an epub in the first instance.

So either Amazon is delivering the Mobi 7 file to all e-readers and apps (I don't use any KF8-specific bells & whistles) or it is delivering my epub to the KF8 crowd. I suspect it's the latter, since the epub is roughly the same size as the download shown on the book detail page on the Amazon store.

I look forward to reading the Wiki; thank you.

[DiapDealer]

Originally Posted by Notjohn

As noted, Kindleunpack shows me only two folders, Mobi 7 and Mobi 8, and the latter includes only a file with the extension *.pub.

Yes, I uploaded an epub in the first instance.

So either Amazon is delivering the Mobi 7 file to all e-readers and apps (I don't use any KF8-specific bells & whistles) or it is delivering my epub to the KF8 crowd. I suspect it's the latter, since the epub is roughly the same size as the download shown on the book detail page on the Amazon store.

I look forward to reading the Wiki; thank you.

Amazon is delivering Mobi to devices/apps that don't support KF8. They're delivering KF8s to the devices/apps that do. No one is getting an "ePub" delivered to them from Amazon. KF8 has a lot of similarities to ePub, but make no mistake: it's NOT ePub. There is no ePub hiding "inside" a file that gets delivered to an Amazon customer. KF8 is still records inside a binary database, just like Mobi. Amazon also never delivers the original source (be it html or ePub) to customers. They deliver old Mobi OR KF8 depending on the capability of the device/app.

As far as KindleUnpack goes: it tries to reverse-engineer the kindlebook in order to recreate (as close as possible) the original source used to create the Kindlebook. As such, it's chosen to recreate that code into the form of an ePub when it's dealing with a KF8 book. It could have just as easily been created from html files, but there's no real way of telling that from the KF8 file. KindleUnpack tries its best, but it's altogether possible for it to create an invalid ePub -- simply because of all the different sources kindlegen will/can build a kindlebook from. Most of the time, it works well (especially if the kindlebook WAS created from an ePub), though.

Now: starting with a kindlebook that's been delivered to an end-user (customer), you get a MOBI, or a KF8. Never both, and never the original source.

So using that retail-delivered kindlebook: if KindleUnpack produces a mobi7 folder AND a mobi8 folder ... then you had a KF8 (and ONLY a KF8) to start with. The ePub that KindleUnpack attempts to reverse-engineer the KF8 book into, is built from the contents of the OEBPS folder (which was extracted/extrapolated from the binary database) inside the mobi8 folder. The mobi7 folder contains only images (if there were any) and can be considered vestigial in this instance.

If KindleUnpack produces ONLY a mobi7 folder from that same retail-delivered kindlebook, well then clearly it was a regular old run-of-the-mill mobi. The extracted markup, images and opf/ncx will be in that mobi7 folder.

Where it gets complicated is when KindleUnpack is dealing with the raw, hybrid MOBI/KF8 output that Kindlegen/KindlePreviewer produces ... the file that authors and publishers submit to Amazon (which is the same file that the KDP conversion process produces if you upload ePubs) ... a file that the end-user never sees. KindleUnpack will extract both the MOBI and the KF8 code into their respective folders. It will also cleave off the original source that was included as part of that hybrid MOBI/KF8 file. That original source will manifest as a zip archive.

[HarryT]

What was it that you used unpack on? The file that "KindleGen" created, or the version of the book that you actually downloaded from Amazon? As has been said, the book that Amazon sends to a customer is NOT the same as the file that KindleGen creates.

[Notjohn]

Originally Posted by HarryT

What was it that you used unpack on? The file that "KindleGen" created, or the version of the book that you actually downloaded from Amazon? As has been said, the book that Amazon sends to a customer is NOT the same as the file that KindleGen creates.

Yes, the download *.mobi from the KDP conversion (Step 6 in the publishing process).

Okay, I think I am beginning to understand what happened here. The Mobi 8 folder contains an epub file, but it is actually the KF8 file reconstituted, and not the epub that I uploaded in the first instance. Is that right?

So when I am told that the converted *.mobi contains THREE books (the one I uploaded plus a Mobi 7 and a KF8 file), either that's incorrect or else I can't prove it from Kindleunpack?

Sorry to be so thick about this.

(And thanks for the reassurance that the *.mobi created on a computer by Kindlegen or Kindle Previewer is NOT identical to the *.mobi created by the conversion process online at KDP. I am glad to know that!)

[DiapDealer]

Originally Posted by Notjohn

Okay, I think I am beginning to understand what happened here. The Mobi 8 folder contains an epub file, but it is actually the KF8 file reconstituted, and not the epub that I uploaded in the first instance. Is that right?

Yes, that's right.

Originally Posted by Notjohn

So when I am told that the converted *.mobi contains THREE books (the one I uploaded plus a Mobi 7 and a KF8 file), either that's incorrect or else I can't prove it from Kindleunpack?

I'm not quite sure I follow, but perhaps it's because I have no experience with what the KDP process provides for you (in the way of a .mobi file to verify) when uploading ePubs. I only know what Kindlegen creates and what ultimately gets delivered to customers.

Originally Posted by Notjohn

Sorry to be so thick about this.

Not at all. The dual-format scenario definitely adds layers of complexity.

Originally Posted by Notjohn

(And thanks for the reassurance that the *.mobi created on a computer by Kindlegen or Kindle Previewer is NOT identical to the *.mobi created by the conversion process online at KDP. I am glad to know that!)

I certainly didn't mean to reassure you of that. As I mentioned, I have no experience with what sort of *.mobi file(s) someone who submits ePubs to KDP gets returned to them for verification. But I suspect they're nearly identical to what kindlegen would produce locally. I'd have no trouble believing the original source isn't included, though.

[Notjohn]

Originally Posted by DiapDealer

I'd have no trouble believing the original source isn't included, though.

Yes, I'm beginning to think that that's the case. Perhaps the confusion (on the part of those who claim three books in the converted *.mobi file) is the result of Kindleunpack's making an epub out of the KF8 version. Since I uploaded an epub in the first instance, I naturally assumed that was my book ... but if that were the case, then where's the KF8 version?

(And the book as shown on the Amazon store has a file size more or less the same as the Mobi 8 / epub returned to me by Kindleunpack.)

[HarryT]

When you create an ebook using "Kindlegen", the generated file DOES include the source. Is that what you're using to create your book, or are you uploading an ePub directly to KDP?

[Notjohn]

Originally Posted by HarryT

When you create an ebook using "Kindlegen", the generated file DOES include the source. Is that what you're using to create your book, or are you uploading an ePub directly to KDP?

Directly to the KDP. Then I download the converted *.mobi for previewing.

Okay, Kindlegen (or Kindle Previewer) does include the source? That would explain where the conventional wisdom originated.

[Hitch]

Originally Posted by Notjohn

(And thanks for the reassurance that the *.mobi created on a computer by Kindlegen or Kindle Previewer is NOT identical to the *.mobi created by the conversion process online at KDP. I am glad to know that!)

That's not what he meant, or, rather, you've misunderstood.

The file that is delivered to the customer is not the same as the file created by KindleGen/KP. The "Step 6" file and the KP/KG desktop or command-line generated file ARE the same.

And, yes, the source file archival copy is included in the final file uploaded to Amazon "for sale," if you either a) upload an ePUB at KDP or b) create a mobi file on your desktop using KG/KP. The archival copy is the ePUB. The K8 copy is the K8 (or, to not confuse matters further, the ePUB emulation that works on mobi devices, in a mobi wrapper). The K7 copy is actually an old-style prc, for all intents and purposes. For that matter, if you uploaded a zipped HTML file, for all I know, that archival copy is retained as well (I don't see why it wouldn't, as for all intents and purposes, it's an ePUB sans OPF and NCX), but I can't speak to that as I've never done it, going straight to fully-made eBooks from the beginning, even when they were MBPC mobi's.

Hitch

[Notjohn]

Originally Posted by Hitch

And, yes, the source file archival copy is included in the final file uploaded to Amazon "for sale," if you either a) upload an ePUB at KDP or b) create a mobi file on your desktop using KG/KP.

This is very strange! The downloaded, converted book file does not contain the original submission, but it (the original submission) is later tucked into the file at an invisible "Step 7"? (Or has Kindleunpack ignored one of the two files, either my original submission or the epub-plus-wrapper as you neatly describe it?)

Originally Posted by Hitch

The K8 copy is the K8 (or, to not confuse matters further, the ePUB emulation that works on mobi devices, in a mobi wrapper).

So the answer to my original question is: Qualified Yes? Amazon is delivering epubs-in-a-wrapper to owners of the Fire and related devices and apps?

[HarryT]

Originally Posted by Notjohn

This is very strange! The downloaded, converted book file does not contain the original submission, but it (the original submission) is later tucked into the file at an invisible "Step 7"? (Or has Kindleunpack ignored one of the two files, either my original submission or the epub-plus-wrapper as you neatly describe it?)

The file created by "Kindlegen" does contain the original source of the book, but this will not be contained in any file that Amazon sends you. You can trivially prove this for yourself: compare the size of the Kindlegen output with what you get back from Amazon - the Kindlegen file will be a lot bigger, because it contains the source code.

So the answer to my original question is: Qualified Yes? Amazon is delivering epubs-in-a-wrapper to owners of the Fire and related devices and apps?

No, KF8 is not ePub. You can think of it as "ePub in a wrapper" if it helps you to do so, but it's not ePub. KF8 is supported by all Kindle apps (PC, Mac, Win8, iOS, Android) and all Kindle readers from the Kindle Keyboard onwards.

[KevinH]

Hi,

To make this clearer, for this discusssion let's ignore the source zip archive stored in one record inside the .mobi file when generated with kindlegen ... and just talk about the format of a K8 (mobi) azw3 file:

- it is NOT simply an epub wrappered into a Mobi

To see this, look at the huge amount of code in KindleUnpack that takes a processed kf8 database file (as generated by kindlegen or downloaded from Amazon) and tries its best to create an epub-like structure from it.

You can enable the "raw" mode to see what the actual pre-processed text inside the .azw3 looks like and you will see all file names are removed, all links are encode by byte position, how svg is extracted, how css is extracted and collected, etc.

You can enable "debug" mode to see all of the processed indices that are used to create entry and exit point offsets used to split that text into separate chapters and fragments. There are also indices that are used to store the NCX and other navigation pieces.

Of course the metadata is stored in the Mobi header and created partially from the opf and partially form other sources.

All of this needs to be "undone" to create an "epub-like" structure from the .mobi record database that an .mobi or .azw3 really represents.

This should make it abundantly clear that a mobi file created by kindlegen is nothing similar to an actual epub just wrappered, it is instead a highly preprocessed database of records and offsets that mimics/extends the original .mobi format.

That said, almost all of the information in an epub has its equivalent in a KF8 mobi and anything supported by the various versions of WebKit should work equivalently in an epub and a KF8.

Hope this makes things clearer.

Ask if you want more details about what the internal structure of a kf8 (or Kindlegen) generated mobi looks like, I would be happy to describe the epub-extraction process in detail.

Take care,

KevinH

Originally Posted by Notjohn

This is very strange! The downloaded, converted book file does not contain the original submission, but it (the original submission) is later tucked into the file at an invisible "Step 7"? (Or has Kindleunpack ignored one of the two files, either my original submission or the epub-plus-wrapper as you neatly describe it?)



So the answer to my original question is: Qualified Yes? Amazon is delivering epubs-in-a-wrapper to owners of the Fire and related devices and apps?