Hopefully an expert on kindle firmware will take a look at the software that handles kfx in order to work out more details of how the new format works. I don't have those skills, but I love a mystery and so I decided to see what I could discover just by inspection of the files. I thought that perhaps some of my experience in reading core dumps in a former life could come in handy.
I am making some progress. I tried theorizing on how the files are structured and wrote code to attempt to parse them based on those theories. I iterated through a large number of failures, but now I think that I have a pretty good overall idea of how the files work. So far I have found three innovations in the new format.
First, the auxiliary kfx files in the sdr directory have a format similar to azw6. Those files start with a CONT (container?) block which has a table of pointers to ENTY (entry?, entity?) blocks. Each of those represents a resource, such as an image file. Logically it is equivalent to a directory of files, each identified by a number.
Second is a new data encoding that can be found in all of the kfx-related files. This seems to be a serialized binary object format, like a binary version of json and similar to
message pack. I did some research, but as far as I can tell the object coding being used is new and unique to Amazon. If not perhaps someone might recognize it from my description.
Each data block starts with a header of e0 01 00 ea. The format has compact coding for numbers, strings, blobs, integers, lists, objects, etc. It relies on the variable-length integer format used for some older kindle data structures.
The main problem with decoding it is that the identification of object and attribute types is by an ID number. In order to make sense of the data you need a mapping of these ids to the corresponding object/attribute name. I have been able to figure out several by inspection, such as book title and author and image height and width, but most are not so obvious. There are hundreds of different ids. Hopefully someone will come across a list of mappings.
Third is the apparent use of encryption to hide the content of the primary kfx file, even for non-DRM protected books. The DRMION format appears to be a list of objects (encoded as described previously), each of which has an encrypted blob and a series of parameters that appear to describe the method for decoding it. I haven't had a chance to research this further yet.
Exploring kfx is like peeling an onion. Decoding one level of the file structure just opens another that needs to be examined.