KFX Format

[baf]

Originally Posted by KevinH

I simply can not get Previewer to actually make a kfx (or kfd) file of any kind no matter which device I pick. Any hints on how to invoke it to see it actually it make a kfx file from an azw3?

KevinH

I use the sample epub from amazon, link is in FAQ – Q8 on that page. I just drag this epub over Previewer and it produces kdf and some other files in a hidden directory and mobi file somewhere in Documents.
Mobi file doesn't have any enhancements.
The file I posted is deleted during conversion process, so you must be quick to capture it

I will have a look at env vars later.

[Doitsu]

Originally Posted by KevinH

I simply can not get Previewer to actually make a kfx (or kfd) file of any kind no matter which device I pick. Any hints on how to invoke it to see it actually it make a kfx file from an azw3?

The .kdf file is generated by the Windows version of Kindle Previewer in the user's TEMP folder. If you enter %temp% in the Run dialog box, you'll find it in a book subfolder of a random UUID folder.

[jhowell]

Here is an overview of what I have been able to discover so far from examining kdf files:


Html and css files are replaced by text with associated formatting instructions in binary data structures. Semantic tags, such as em and strong, produce no difference in the result other than the formatting they represent.

The possible formatting instructions are based loosely on css properties, but changed and somewhat simplified.

The end result is much more like a reflowable pdf than an html document.

Also, there are no soft hyphens or other hyphenation indicators added to the text.

[jhowell]

Originally Posted by HarryT

We did know that one of the goals of the KFX format was to supply device-specific formatting - eg image resolutions tailored to the device.

It is still unclear to me how many variations are in use. So far I have only seen two: low resolution B/W jpeg-xr for e-ink kindles and high resolution color jpeg for everything else.

One thing I have noticed is that the 'attachables' files, containing book images, are limited to about 3MB each. Books with lots of images will have several of these files. When downloading one of these books with the newer kindle apps, the download progress bar will quickly turn green. At that point the app will allow the book to be opened, even though image files are still being downloaded. This seems like a useful feature for those with slow internet connections.

[JSWolf]

Originally Posted by baf

I also played with some temporary files created during conversion. That gives an idea about optimization process.
For example have a look at changes introduced to the original xhtml file, while converting epub to enhanced mobi.

Your URL leads to a 404. Please repost.

[baf]

Originally Posted by JSWolf

Your URL leads to a 404. Please repost.

Updated my post

[darryl]

Originally Posted by jhowell

Here is an overview of what I have been able to discover so far from examining kdf files:


Html and css files are replaced by text with associated formatting instructions in binary data structures. Semantic tags, such as em and strong, produce no difference in the result other than the formatting they represent.

The possible formatting instructions are based loosely on css properties, but changed and somewhat simplified.

The end result is much more like a reflowable pdf than an html document.

Also, there are no soft hyphens or other hyphenation indicators added to the text.

Thanks for the overview. It is nice to see that some progress is being made. This makes me wonder whether there is a completely new rendering engine at work or perhaps simply a modified one to incorporate dealing with the new formatting instructions.

[baf]

Originally Posted by KevinH

baf, can you get a dump of the environment variables that are set during the conversion to kfx? The internal Epub conversion jar seems to look for AMZN_ prefixed environment variables in places but so it might help to see what is getting set.

KevinH

I don't see any such variables.
Previewer runs following command to convert epub to kdf. Run it from lib/fc directory. Then watch /tmp directory for resulting files. Also notice the files that are created and then deleted.
This applies to OS X, should be similar for windows.

jre/bin/java -cp lib/EpubToKFXConverter-1.0.jar:lib/* com.amazon.kfxconverter.app.KFXGenApp -libDir . -tmpDir /tmp -outDir /tmp -inputFile sample.epub -locale en -amzncreator "Kindle Previewer 3.0.0"

The command takes following parameters. I don't see any profiling options here though:

Usage: <main class> [options]
  Options:
    -amzncreator
       tool version (optional)
    -compressionType
       Compression type
  * -inputFile
       Input file
  * -libDir
       Base directory where all the required libraries are present
  * -locale
       The language for Kindlegen logs
  * -outDir
       Output directory where mastermobi is stored
    -outFileName
       Output file name for .mobi, if not given the input filename is taken
  * -tmpDir
       Temp directory where kdf is stored

[DaleDe]

Originally Posted by jhowell

The previewer produces a new file type, KDF, which is basically KFX data in an sqlite3 database instead of a KFX container. I wonder if that is something specific to the previewer software or if this is an intermediate format produced as part of their production tool chain. I'm learning quite a bit from examining these files.

Please record your knowledge in our wiki.

Dale

[KevinH]

Hi jhowel,

Originally Posted by jhowell

The end result is much more like a reflowable pdf than an html document.

Yes, that explains the fixed height and width properties throughout. My guess (complete wag) is they map a reflowable epub into a size fixed to a specific device (or family of devices), handling automatic hyphenation (since the boundaries are now known), and then use webkit to effectively "print" it to a pdf-like format, and deliver that to your device. Kerning, ligatures, font metrics (known fonts!), hyphenation positions, Dropcaps, etc can all be worked out in advance in a fixed layout and converted into a form suitable for viewing. Then sell the whole pile as "Enhanced formatting" which is similar to what can be achieved in a finished format like pdf.

KevinH

[KevinH]

Hi jhowell,

Does the generated have any drmion atached to it or is it DRM free? If DRM free, what chances do you think someone might have to convert this reflowable pdf-like beast back to pure html and css?

Thanks,

KevinH

[jhowell]

Originally Posted by KevinH

My guess (complete wag) is they map a reflowable epub into a size fixed to a specific device (or family of devices), handling automatic hyphenation (since the boundaries are now known), and then use webkit to effectively "print" it to a pdf-like format, and deliver that to your device. Kerning, ligatures, font metrics (known fonts!), hyphenation positions, Dropcaps, etc can all be worked out in advance in a fixed layout and converted into a form suitable for viewing. Then sell the whole pile as "Enhanced formatting" which is similar to what can be achieved in a finished format like pdf

I don't think it's quite that bad. They are still using a reflowable document model, it's just not html.

A lot of what got added in that intermediate html file shown by baf doesn't appear in the kdf output. It looks to me that they are determining the final styling result of css for each html element and then translating that to their own simplified binary document model.

I don't see any sign that they are pre-rendering to specific devices. That would be difficult given all of the possible font-face/size combinations and screen sizes for all iOS/Android devices. Also, I have been able to move non-device locked kfx files between devices of different resolutions and they rendered just fine.

Drop caps appear to be handled by using their own non-standard style properties created just for that purpose.

The document text doesn't appear to be altered at all. I think that kfx hyphenation is handled by the renderer on the user's device using a dictionary.

[jhowell]

Originally Posted by KevinH

Does the generated have any drmion atached to it or is it DRM free? If DRM free, what chances do you think someone might have to convert this reflowable pdf-like beast back to pure html and css?

There is no sign of DRM or encryption in the kdf files produced by the previewer.

In kfx format some information, such as metadata, is in an unencrypted container and the format of that data matches the format of the metadata found in a kdf file.

In kfx all of the actual text and document layout is in an encrypted container. In kdf that data is all in the clear and is encoded in the same binary format as the metadata. (The encrypted kfx container almost certainly holds the same data format, once decrypted.)

The data format is a representation of a nested data structure, sort of a binary version of json. I previously worked out some of the details and it is discussed earlier in this thread and there is even some code that partially handles it already in calibre. (See the PackedIon class.)

I think it is quite doable to turn kfx/kdf back into html, but it would take some work to figure out how their document model maps back to standard html. It should be possible to do this preserving visible content and formatting, but some details, such as the exact css classes used in the original document will be lost. Of course the drm used for kfx books would still be an issue.

[Notjohn]

>I think that kfx hyphenation is handled by the renderer on the user's device using a dictionary.

I have a new Paperwhite and a new Fire; the latter shows auto hyphenation but the former doesn't.

Also I don't understand how there could be a fixed page width in a reflowable book when the user has the ability to change the font size.

I'm glad to see this discussion continued!

[jhowell]

Originally Posted by Notjohn

I have a new Paperwhite and a new Fire; the latter shows auto hyphenation but the former doesn't.

If you see this difference for the same book then the most likely cause is that book not being in KFX format on your Paperwhite. You can verify by connecting to a computer via USB and finding the book in the documents directory.

If the book is in fact downloaded in KFX format on both devices and it renders differently I would like to know more about that.

Originally Posted by Notjohn

Also I don't understand how there could be a fixed page width in a reflowable book when the user has the ability to change the font size.

That was speculation by KevinH based on the contents of the intermediate file format posted by baf. I agree with you that the contents are reflowable, not fixed.