KFX Format

[pdurrant]

The posted 'HTML' clearly sets a fixed page width, but I suspect that's just a way to give a number for all the other measurements to be calculated against. I'm sure that when displayed that fixed width gets scaled according to the device and the font size.

[KevinH]

In case anyone is interested "procyon" (a good java decompiler) exists and is a free download.
It is available here: https://bitbucket.org/mstrobel/procyon/downloads See that site's wiki page for info on how to run it.

From a quick glance at things and the class names alone it appears that there is an internal copy of gson (google's java object to json and back serializer/deserializer) being used. There is also a large list of common (all?) html tags terms. Perhaps all of the html elements have been replaced with offsets into a table/map of html elements? Again this is just based on a quick glance before heading off to work this morning. Given gson is used (we can see its source code online since it is open source), perhaps the kdf sqlite3 fragment "blobs" are in fact serialized java objects (serialized to json?).

KevinH

[jhowell]

Originally Posted by DaleDe

Please record your knowledge in our wiki.

I am going to hold off on doing that. I lot of what I have is speculation, subject to change. I would rather spend my available time investigating at this point.

Most of what I have found out has already been posted in this thread. If someone else wants to take a stab at organizing it for the wiki that is fine with me.

[baf]

Apart from decompiled java, a good source of information about ION format is library enigmatically called libshared (part of Previewer).
I am attaching mapping of ids to names for ION types and properties. May be useful, if you don't have it yet (I've seen some of it used in kfx.py).

[KevinH]

baf,

Nicely done!

This is a huge help towards allowing something else to read/write this format.
Thank you.

KevinH

Originally Posted by baf

Apart from decompiled java, a good source of information about ION format is library enigmatically called libshared (part of Previewer).
I am attaching mapping of ids to names for ION types and properties. May be useful, if you don't have it yet (I've seen some of it used in kfx.py).

[KevinH]

baf,
Are you sure about your last Ion symbol Datagram value? The reason I ask is that on the java side these are just a straight enum with DATAGRAM being last which would give it a value of 13, and tid_none and tid_eof are not part of the enum at all.

BTW: anyone playing with procyon needs to use a case-sensitive file system. Many of the classes are given the same name with the only difference being case (ie. A.class and a.class) in the same directory. These unpack over themselves and you lose the information. So if you are using Windows or Mac to play with procyon and any jar you need to be aware of that. On a Mac, you can of course take any usb key or spare drive and use DiskUtilities.app to erase and format it as a Mac OS X Journaled Case-Senstive filesystem if you want to play around with procyon. Linux is case-sensitive normally. Not sure what you can do on Windows.

KevinH

[jhowell]

I suspect that the enumeration with names that start 'hn' will be the most useful.

[baf]

Originally Posted by KevinH

Are you sure about your last Ion symbol Datagram value?

This is not a typo, but treat it with caution, as I don't know what the purpose of this data is. Also note that not all "tids" match "types" in my table. On the other hand my "tids" fit hn_ enumeration scheme, mentioned by jhowell.

Originally Posted by KevinH

BTW: anyone playing with procyon needs to use a case-sensitive file system.

I use JD-GUI on OS X

[jhowell]

You can save the book.kdf file created by the previewer for later use. You need to also save the 'res' directory associated with it, if any. It holds the images and fonts of the book.

You can open the KDF book with the previewer using the control-O command, at least under Windows. You need to navigate to the directory where you saved it and manually enter the name book.kdf in the open dialog. This works even though .kdf isn't shown as a supported format.

To re-open properly the file name must remain book.kdf. This file name seems to be hard coded into the previewer.

[Notjohn]

Curiously, my most recent book (an omnibus of five previously published e-books) does NOT have Enhanced Typesetting enabled, though the included books all do. I wonder, then, if KFX is not being added in the publishing workflow but in some later operation? (I bought the book, as I did the earlier ones, though the earlier ones were all published before KFX was introduced.)

And for what it's worth, I commented on KFX on the KDP forum this morning, only to have the post sent to moderation. I wonder if "KFX" has been added to the list of forbidden words, like "description"?

Later: Oh my goodness (as Donald Rumsfeld said in another connection), I just republished the book (yes, I found a typo), and I see that Enhanced Typesetting is now enabled. That was fast.

[jhowell]

Originally Posted by Notjohn

And for what it's worth, I commented on KFX on the KDP forum this morning, only to have the post sent to moderation. I wonder if "KFX" has been added to the list of forbidden words, like "description"?

I think that Amazon wants to have everyone think about the enhanced typesetting feature and be unaware of the KFX file format associated with it. Even the new Kindle Previewer beta doesn't produce a KFX file that you can sideload to your own device. (The hidden KDF file produced is not sufficient.)

Originally Posted by Notjohn

Oh my goodness (as Donald Rumsfeld said in another connection), I just republished the book (yes, I found a typo), and I see that Enhanced Typesetting is now enabled. That was fast.

It is still mysterious to me why some books convert to KFX/KDF and others don't. I have been playing with the Previewer beta and while most books I've tried show Enhanced Typesetting (and produce a KDF) some fail to do so. Unfortunately, the previewer doesn't even tell you that KDF conversion failed. You need to use the ctrl-I (information) screen to see whether or not it shows "Enhanced Typesetting supported" to find out.

I have found no way to discover why a book was rejected. The conversion process produces some logs, which are immediately deleted. Even when I have been able to capture them I have yet to find an indication of what went wrong. Amazon has really failed to be helpful here. This leaves only trial and error to solve enhanced typesetting problems. (At least trial and error is easier with the previewer.)

[baf]

Originally Posted by jhowell

It is still mysterious to me why some books convert to KFX/KDF and others don't.

The reason is usually in one of the logs. I rsynced temp directory in a loop to get all output.
Creating kdf file is a very complicated, multi-staged process. Input html files are cleaned, parsed, individual tags, styles are extracted, images are optimised, mobi files are created and then parsed again. The process involves multiple applications, many temporary files and directories are created and deleted.
If any of the stages fails kdf file is not produced and previewer renders mobi file without enhanced features. I suppose most of the errors are due to conservative approach – if there is a chance that parsing might break original layout of the document, the process fails.
I didn't have time to test many documents. Some of the errors were as simple as: "The Language given is not supported currently", other more cryptical: "Not exact match - YJ=ContentFormat:YJCANONICAL, ContentType:IMAGE, TextChar:null at md pos 3,Mobi=ContentFormat:MOBI8, ContentType:TEXT, TextChar:T at 283"

[jhowell]

Originally Posted by baf

The reason is usually in one of the logs. I rsynced temp directory in a loop to get all output.

Thanks for the tip. I set up syncing of my temp directory and it is easier to see what is happening in the logs.

The KDF generation process is complicated! (Rube Goldberg comes to mind.)

[jhowell]

Amazon has another tool, the Kindle Textbook Creator, that can create interactive multi-media textbooks starting with a PDF file. It packages books for uploading to Amazon in Kindle Package Format (KPF) and these .kpf files can also be opened by the new Previewer 3.0 beta.

After trying the Textbook Creator out I examined a KPF file and found that it is just a zip file containing a KDF book, similar to what the Previewer beta creates for enhanced typesetting. I'm guessing that once the KPF is uploaded to Amazon it will be delivered to customers as a KFX book.


I experimented a bit and discovered that you can turn an enhanced typesetting book created by the Previewer beta into a KPF file that you can re-open later using just a few steps:

• After converting a book using the Previewer locate the directory within your temp directory that holds the results. It should have 'book' and 'conv_metrics' subdirectories.
  
• Rename the 'book' directory to 'resources'.
  
• Zip the 'resources' directory using your zip utility of choice. You want a zip file that contains that directory name in it.
  
• Rename the zip file to have the extension '.kpf'.

You can now open the KPF file using the Previewer.

[jhowell]

Core to KFX is the ion data representation. It is a means of encoding a complex data structure into a compact binary format. It includes representations for numbers, strings, symbols, and various types of structured data.

The separate ion data structures that make up a book are called fragments or entities. In KDF files produced by Kindle Previewer 3 they are placed in an SQLite database.

In KDF the fragments form a hierarchy containing the book content:

• document_data contains a list of sections in reading order.
• Each section corresponds to an html file in the source EPUB. It has a page template and a reference to a story.
• A story contains a list of content. Content types are based on HTML: container (nested div), text (div, p, h1, etc.), image (img), horizontal_rule (hr), list (ul, ol), listitem (li), table (table), etc.
• Sets of formatting instructions are grouped into a style that has properties that are based on HTML attributes and CSS properties. (For example, "background-color" becomes "fill_color".)
• There are also fragments containing metadata, and navigation (toc, page numbers, locations, and positions).

KFX appears to contain pretty much the same contents as KDF, just packed in a proprietary container format. (As least the visible parts of KFX, not protected by DRM. I assume that the hidden data also corresponds.)