KFX Format

[jhowell]

After installing the latest PW2 firmware with enhanced typesetting I downloaded a book that I had previously purchased so that I could see the new features in action. I looked in my Kindle's documents folder and found the book there with a 'kfx' extension instead of 'azw' or 'azw3'. Does this indicate a new file format for enhanced typesetting Kindle books?

[Nate the great]

Both of the ones I checked were AZW3, not KFX.

On the other hand, neither are showing the improved typography that Amazon promised, so perhaps that is a clue?

[odamizu]

It does indeed look like Amazon has introduced a new ".kfx" format to go with the enhanced typesetting engine:

http://allesebook.de/firmware-softwa...-kindle-65388/

English translation: https://translate.googleusercontent....OhpVElocmuTYvw

Among other things, the new format appears to include soft hyphens.

[jhowell]

Some interesting strings from inside a kfx book:

{key:"kfxgen_package_version", value:"PackageVersion:YJReaderSDK-1.1.67.2 Month-Day:06-18"},
{key:"kfxgen_application_version", value:"6.24.1.0"}

From this I'm guessing that Amazon has a kfxgen tool that they use internally to produce these files.

ETA: Perhaps this will become available more broadly if Amazon updates their Kindle Previewer tool to support enhanced typesetting.

Edit 2: Another interesting thing is that the DRM strings are present in the file even for supposedly DRM-free books. It looks like Amazon may be treating this new format as more proprietary than their older ones.

[jhowell]

I did some tests with my PW2 to experiment with the possibility of side-loading a book in kfx format. To do this I purchased and directly downloaded a free book to the kindle. I then switched into airplane mode and using USB I saved a copy of the book's main kfx file and the associated sdr directory. I deleted the book from the kindle and then made several attempts to side-load the book back to it, using a different file name each time.

I was able to successfully side-load. For this to work I needed the main kfx file, a directory of the same name with the suffix of '.sdr', and a subdirectory named 'assets' containing a small file named 'voucher' copied from the original download. This was sufficient to have the book show up as a personal document, but without a cover thumbnail, images, or a table of contents.

I then added the 'metadata.kfx' file that had originally been contained in the assets directory. That made the book functional again with a proper cover thumbnail, internal images and a working table of contents.

Finally I added an 'attachables' subdirectory within assets containing a file named 'CR!FYVV6BXDES0YQ4MCX4CG93WB1Z0B.kfx'. This didn't make any apparent difference in the working of the book in my quick test.

That last file contains a number of high quality jpeg-xr images from the book. I don't understand how the book worked without them. Perhaps there are lower quality images elsewhere as a fallback or perhaps there were cached images somewhere on the kindle.

I only have one kindle so I am unable to test whether this process would have allowed to book to be side-loaded to a different kindle. I suspect that it would not have worked due to DRM.

All of this gives me hope that if the kfx format is ever understood well enough, it may be possible to create and side-load non-Amazon books in kfx format and have them rendered with enhanced typesetting.

[odamizu]

Thank you jhowell for all this testing and taking the time to report your findings.

It will be interesting indeed to see if Amazon releases a kfxgen tool or does all the conversion internally.

What does everyone think about the future of mobi and azw3? Do you think Amazon will support three?!? formats going forward, or will they deprecate at least one? I don't mind losing mobi, but at this point I'm feeling rather attached to azw3.

[eschwartz]

They might kill off MOBI as soon as they feel comfortable abandoning all K2/DX* owners. Probably only a few years away.
AZW3 is currently the bleeding edge as far as the K3/K4/KT know, as well as however many never-connected-to-WiFi Pw1/PW2/PW3/KV's there are. All seven of which Amazon are not going to cut off support for, going by their track record.



* -- I don't know if there are any owners of an Original Kindle, though.

[kovidgoyal]

While I dont have a Kindle capable of enhanced typesetting myself, here are my guesses about KFX:

1) It is most likely a modified form of azw6 -- azw6 is a format that appeared a few months ago, where a normal azw3 file gets its high-res images downloaded separately from the text and put into the sdr folder. This was likely a bandwidth saving feature. https://bugs.launchpad.net/calibre/+bug/1429007
Alternately, if they have finally gotten tired of Palm Databases, it could be a modified for of AZK -- their special snowflake for iOS.

2) KFX files are likely to be "pre-rendered" to some extent, such as having hyphenation pre calculated and added to the text as either soft-hyphens or html tags. This is similar to what Kobo does with kepub.

3) I would not be surprised if amazon is encrypting all kfx files regardless of whether the original has DRM or not. This would be consistent with their "boiling frog" approach to closing off their ecosystem. Over the years they have gradually made their ecosystem more and more closed and hoped that if they do it slowly enough people wont notice. Cases in point: No email delivery of azw3, the completely unnecessary personal document category which tries to separate sideloaded books from purchased books, azw3 itself which was a completely unnecessary obfuscation of EPUB. It should be easy to check if kfx files are encrypted -- just do a binary search for strings you know appear in the text (preferably short ones so that soft hyphens dont interfere)

[quiris]

Originally Posted by kovidgoyal

It should be easy to check if kfx files are encrypted -- just do a binary search for strings you know appear in the text (preferably short ones so that soft hyphens dont interfere)

They are indeed encrypted. Even free books from Public Domain. The only "readable" part of the binary blob is the beginning of file (DRMION, ProtectedData etc.). Check a screenshot in the first post in this thread.

[AcidWeb]

After observing Amazon movements for last few years another move to close ecosystem even more is not a surprise...

[KevinH]

Hi,

In case it matters. "drmion", based on a simple Google search, is one variation of a Permutation Cipher that permutes the positions of bytes/characters in each set of K characters to simply protect the data (not really encrypt it). To decipher it you either need the key (or its key length and a piece of known plaintext). It is not a strong cipher.

Of course, the "DRMION" and "ProtectedData" shown in the picture in an earlier post may have nothing whatsoever to do with the drmion permutation cipher (there was an AES/CBC string there as well according to the picture as well) but if someone had access to the full .kfx file and got the same version of the .azw3 or mobi file) you may be able to figure out if they are simply using a permutation cipher to confuse the issue.

My 2 cents ...

KevinH

[eschwartz]

According to @shalym, there is no DRM:

Originally Posted by shalym

Ok...I just tested with a DRM free book that is in the .kfx format, and I was able to transfer the book from my Voyage to my pw3 and have it transfer correctly. I had to move both the .kfx file and the .sdr folder associated with the book.

There were a couple of files that were not able to transfer. The error given was that the filename was too long. (This is using Win 8.1) Those files being missing didn't affect the functioning of the book, at least not as far as I could tell. Hyphenation still works, and the enhanced spacing at larger font sizes works too. I can't judge the kerning and other stuff, as I really can't see that.

Shari

If the book works on two different devices, it cannot be DRMed (I suppose Amazon *could* have moved backwards to per-account keys), so I am going to assume quiris' "unreadable blobs" are just the "new format we don't understand yet" part.

Thanks for all the pessimism.



EDIT2: s/encrypted/DRMed/g because of nitpicking.

[shalym]

Originally Posted by eschwartz

According to @shalym, there is no DRM:



If the book works on two different devices, it cannot be encrypted (I suppose Amazon *could* have moved backwards to per-account keys), so I am going to assume quiris' "unreadable blobs" are just the "new format we don't understand yet" part.

Thanks for all the pessimism.

I just tried adding the book to an unregistered Kindle by using the same method, and it worked. It would appear that DRM is not added to all books in the new format.

Shari

[eschwartz]

Unlikely possibility confirmed averted!

[KevinH]

So, it might just be obfuscated using the drmion cipher. If someone can pm me a link to all of the pieces needed to side load an kfx ebook and a link to the same ebook not in kfx format, I'd be happy to look at reversing it.

KevinH