KFX Format

[shamanNS]

Support for just adding (so, no metadata editing & no conversion) books to Library should be trivial to add, but it would be next to useless since Calibre could not read book title and author

[HarryT]

Originally Posted by shamanNS

Support for just adding (so, no metadata editing & no conversion) books to Library should be trivial to add, but it would be next to useless since Calibre could not read book title and author

You would of course need to add it as a ZIP file (or some other container format) since, as you know, a book in KFX format is not a single file; it's a file with an associated folder of auxiliary files.

[DiapDealer]

I keep thinking that at some point, an author is going to want to make use of the Kfx format AND offer it truly DRM-Free. Will Amazon deny that request, I wonder?

Has anyone looked at Amazon's half-assed font obfuscation techique to see if it might be relevant?

[darryl]

Yes. The thought had occurred to me too. I have come across a couple of authors who pride themselves on offering drm free books partly because it allows (or did allow prior to kfx) for easy conversion and reading on a device other than an Amazon one. It is likely of some importance to them if they go exclusive with Amazon. From memory I think Hugh Howey may be in this category.

I suspect Amazon's answer at the moment will be as simple as ebooks still being available in kf8 and older formats.

[Barty]

Afaict DRM free to amazon means there's no limit to number of kindle device you can download the book to. Does not mean unencrypted file format.

[HarryT]

Originally Posted by DiapDealer

I keep thinking that at some point, an author is going to want to make use of the Kfx format AND offer it truly DRM-Free. Will Amazon deny that request, I wonder?

There are DRM-free KFX books - this has already been demonstrated. The fact that it's a binary file format doesn't automatically imply that it has DRM.

[eschwartz]

Nor does the indication that the file format may be obfuscated, imply that there is DRM as such.

[darryl]

That there is likely no drm on some kfx ebooks has been demonstrated earlier by people loading the same downloaded kfx files on different Kindles. The problem seems to be the obfuscation, though not itself DRM, prevents even non-drm files from being converted. If, as an author, you go exclusively with Amazon and opted for no drm owners of epub readers have hitherto still been able to buy the book from Amazon and convert. This of course remains the case with kf8, but not kfx.

[jhowell]

It is pretty clear that encryption is being applied to KFX books. What if the same encryption algorithm is used for DRM-restricted and for "simultaneous device usage unlimited", with the only difference being restriction checking of the decrypted book within the Kindle software? If this is the case then there is no real difference between "DRM" and "obfuscation" for KFX.

[darryl]

I'm sure we will find out. It's just a matter of when.

[tomsem]

Here are a few experiments that should be done:

1 - zip up kfx and ancillary files for a 'DRM free' book and put them on a device registered to a different account. Does it load up properly? (I believe I saw that someone has checked that this works with devices registered to same account) If yes, then there is no DRM worth that term in the way (and since publishers have chosen 'drm free' books, expectation has been set that it is demonstrably drm free in exactly this sense).
2 - apply standard reverse engineering techniques. Do byte patterns reveal any known compression and or encoding artifacts? It seems unlikely that they'd develop something wholly proprietary or 'uncrackable' (see point 1) just to frustrate Kovid. Rather it is probably something they found more efficient and reliable than alternatives, given the requirements.
3 - someone should apply the same amount of effort to cracking what can be found on Fire tablets (and of course do the 'DRM test' of step 1). The layout of files is different, much different. May be harder, or may be easier. Or maybe comparing side-by-side will reveal some design intent and other clues.

I can't do Kindle investigation without buying a new Kindle, which I have no plans for at the moment.

[eschwartz]

1) It has also been tested -- successfully -- on a second, unregistered device.

That is why I keep pointing out that it is obfuscation, not "real" DRM.

[DuckieTigger]

Originally Posted by eschwartz

1) It has also been tested -- successfully -- on a second, unregistered device.

That is why I keep pointing out that it is obfuscation, not "real" DRM.

I am pretty sure Amazon has an optional 2nd layer of real DRM ontop of the obfuscation for those books that demand DRM.

[eschwartz]

I didn't realize that it was actually necessary to point that out...

[tomsem]

We don't know if it is obfuscation yet, in the strict sense (as in 'rendering it obscure, unintelligible, or unclear'). It is as plausible to be the logical outcome of the technical implementation chosen, exclusive of any intent to obscure. For example, they may have come up with something that is smaller to send over the internet, or which loads from storage into memory more quickly than KF8.

We actually have a pretty good idea of what is in the file: text plus 'markup', and maybe a cover image (we should pick things without many images in them). Moreover the text is known to us (we can read it, or in some cases, go to gutenberg.org and download it), and we can compare the source size to the mobi-ized size to the kfx size and at least learn something more about what is going on.

For example, I just downloaded an .azw3 file from my account page and zipped it. the .zip file is about 10% smaller (694KB vs original 768KB). When downloaded to my Paperwhite (2012, no typography update) the .azw3 is about 3% smaller than the .zip file. So it looks like there might be some sort of compression applied just as part of sending it to the device.

On my Fire HD6, I downloaded the same book, and took a peek with ES File Explorer. Its Book folder has 3.2 MB. Inside that is a .prc file which I presume is the main file. It is 750KB, more or less the same size as the .azw3 file downloaded from account page.

But this is just a 'regular' book.

I have another book that is legitimately 'kfx': "meditations" by marcus aurelius (DRM free). On my ancient Paperwhite it is 274KB (azw3). On Fire, the corresponding Books folder has 134KB. There is no .prc. Instead, there's a folder in there with 'kfx' in the name, and there is 100KB of stuff in it. There are 3 files about 30KB each, and the other 8 make up the balance of 9KB. But there is a huge difference between 274 and 134. Now this particular book has images in it, which might somehow account for this (I'm guessing the images are getting especially compressed as part of KFX processing). But is this the explanation? I need to find a text-only book to eliminate this variable, compare the kfx folder contents, and learn a bit more hopefully.

The book is I think derived from gutenberg source, and divided into part 1 and part 2. The gutenburg text file is 415KB, zipped it is 152KB. So, half of that is 75KB, and so the 134KB seems reasonable if you add in heavily compressed images, and 274KB with less heavily compressed images.

But I need to find better experimental subject: a 'real', text-only kfx. On the surface it would appear that in some cases, KFX uses half of the storage of AZW3. That alone would be reason enough for Amazon to choose to adopt a new approach. It would save them some money they now spend on bandwidth.

Interestingly, downloading it from my account page yields a azw3 file that is 846KB, quite a bit larger. Okay, so I ran Kindle Unpack and compared the image folder sizes. Yup, 123KB (off of the Kindle device) vs 696KB (from account download). So at least that accounts for size differences between device and account page downloads: image compression, vs no image compression.

Naturally I'm curious to know if someone could do a similar comparisons involving Kindle (as opposed to Fire) KFX.