KFX Format

[JSWolf]

Originally Posted by AnemicOak

It kind of was. Didn't they have to be physically sent in to have the upgrade done (IIRC)?

Yes, the 500 has to be sent in to be updated. Sony paid for the shipping both ways. But Sony did it so in the end, every Reader Sony released supported ePub (as long as you sent in your 500 to be updated).

[odamizu]

Anyone care to speculate on the likelihood that Amazon will start sending KFX when one uses the "download and transfer via USB" option? (At this point, I'm still getting KF8 when using this method with Voyage and Oasis.)

Also, any thoughts on the likelihood that we'll be able to convert KFX to other formats in the foreseeable future?

Thanks for any insights. I got burned by B&N, and now that I've settled comfortably into the Kindle universe, I'm afraid it's going to go pear-shaped.

[darryl]

I'll speculate, but it is only speculation, and there are many others better informed.

Yes, I do think at some stage Amazon will start delivering kfx via the download and transfer by usb option. I have seen no indication of this, but it does make sense. kfx is now Amazon's chosen format. Many features now rely on kfx. Amazon could have gone another path, but chose not to, for whatever reasons. It is understandable that Amazon would want to give their readers the best reading experience and provide all features, which is why I think it is just a matter of time.

Yes, I do expect that the kfx format will eventually be reverse-engineered and the encryption and if necessary drm "broken". It has simply not been necessary whilst kf8 and other understood formats are still readily available. I cannot guess a timeframe. It would not surprise me to find out that most or even all of the work has been done. However, given the DMCA anti-circumvention provisions and similar laws throughout the world, packaging and releasing such a tool may be problematic. It may circulate privately for some time, though I expect it would eventually make its way to the public. A kick starter or similar type project would be interesting, but probably fall afoul of Amazon's litigation department.

My advice, for what it is worth, is to buy a PW1 or earlier Kindle which supports kf8. I think the earliest was probably a Keyboard, which I think was 3rd generation. This is the most unlikely candidate for kfx support. However, the consensus view here seems to be that Amazon is unlikely to add kfx support to any further devices, so a PW1 is a good bet.

[jhowell]

Originally Posted by odamizu

Anyone care to speculate on the likelihood that Amazon will start sending KFX when one uses the "download and transfer via USB" option?
...
Also, any thoughts on the likelihood that we'll be able to convert KFX to other formats in the foreseeable future?

I agree with darryl's reasoning above on this.

You can read more speculation in the Will K2PC 1.19 and KFX alter your ebook purchasing habits? thread, especially in the last few pages.

[lkmiller]

My completely inexpert musing on this:

Yes, I agree download and transfer will eventually give kfx for those devices capable. As others have suggested, it could be a zip file that's extracted somehow. I think the PW1 is fairly safe from kfx. I imagine the PW1 already would download kfx directly if it could handle it. I don't have a PW1 or any older Kindles myself and would probably stop buying from Amazon if that became the only option to get my hands on non-kfx books. I don't think it will come to that.

I think that kfx coming to Kindle for PC is actually a good thing for getting both the format and its DRM puzzled out. Not only because it's more urgent, but because it's simply easier (for those with the know-how and tools) to see how Kindle for PC handles it.

I'm cautiously optimistic that by the time they force K4PC 1.19 on us and/or start delivering kfx via usb transfer, that the calibre folks as well as Alf & co. will have things sorted.

[Pajamaman]

Originally Posted by darryl

Yes, I do think at some stage Amazon will start delivering kfx via the download and transfer by usb option. kfx is now Amazon's chosen format.

Makes sense.

Originally Posted by darryl

My advice, for what it is worth, is to buy a PW1 or earlier Kindle which supports kf8.

As long as they don't update the PW1 FW to support KFX, then I agree the PW1 is the best bet. Even if the device breaks, I presume you can still download it via PC.

The other issue is K4PC. If the KFX version becomes suddenly mandatory, then we are seeing a tightening of control. The same with USB transfer.

But so far, Amazon seems content to leave holes in their garden for those that wish to use it.

[odamizu]

Thanks for all the speculations.

Assuming programmers make it possible for the rest of us to disinfect and convert KFX (and many, many thanks to these wonderful people), I'm curious what one gets when one "unpacks" a KFX.

@jhowell kindly listed the files that make up a KFX with a quick description of what they contain. How close is this to EPUB? E.g., if someone familiar with EPUB were to gain access to the main container, would they find something recognizable and editable? What's in the resource container and metadata container?

Originally Posted by jhowell

• B00UZSTXL4_EBOK/amzn1.drm-voucher.v1.cd614ffd-319f-4b9e-99b7-238c6fe78f1e.voucher (DRM voucher)
• B00UZSTXL4_EBOK/B00UZSTXL4_EBOK.azw (encrypted main container)
• B00UZSTXL4_EBOK/CR!6AS14VN94H79K68RG9EKT09T41N0.azw.res (resource container)
• B00UZSTXL4_EBOK/CR!SNRJN8W7KX07Z8MVGN4AMWXSFR3E.azw.md (metadata container)

I'm also concerned by @kovidgoyal's assessment a few months ago and wondering if there's anything new to suggest the future is not so bleak:

Originally Posted by kovidgoyal

From what I have been able to deduce, KFX contains images and probably text that is pre-processed/changed to work for the particular device the KFX is sent to. As such it is fairly pointless as an ebook format, in that it is not suitable a source for conversion to another ebook format.

[jhowell]

Originally Posted by odamizu

How close is this to EPUB? E.g., if someone familiar with EPUB were to gain access to the main container, would they find something recognizable and editable? What's in the resource container and metadata container?

As an analogy for Java programmers, EPUB is like source code and KFX is like compiled bytecode. There is an equivalence, but the compiled code looks nothing like the source code that produced it.

For most e-books there is a fairly straightforward conversion to KFX, which could be reversed without anything essential to reading the book being lost. It would be equivalent to developing a decompiler to turn bytecode back into source code. A difficult, but doable project.

Added: In comparison KF8 is like EPUB repackaged. It is far easier to reverse the process. Even if conversion from KFX becomes available, conversion from KF8 will be superior as long as the format continues to be obtainable.

Originally Posted by odamizu

I'm also concerned by @kovidgoyal's assessment a few months ago and wondering if there's anything new to suggest the future is not so bleak:

I think that Kovid's statement is based on early speculation on the exact content of KFX files. There are multiple variations of a the same book produced for different devices, but as far as I can tell there are only three possibilities that only differ in the handling of images: SD gray scale, SD color, and HD color.

Added: Something similar was also introduced for KF8 a while ago, the azw6 HD image container.

[Pajamaman]

Am I correct in thinking that the DRM on KFX files is currently uncrackable. And that therefore it will only be possible to decompile KFX files without KFX?

You've already managed to compile KFX files from other sources, so I presume it's doable to reverse the process and decompile DRM-free KFX. But laborious.

[jhowell]

Originally Posted by Pajamaman

Am I correct in thinking that the DRM on KFX files is currently uncrackable. And that therefore it will only be possible to decompile KFX files without KFX?

I assume that you meant "without DRM". The DRM currently on KFX books delivered by Amazon prevents most of the content from being accessed. It will not be possible to turn them into anything useful without removing the DRM first.

Originally Posted by Pajamaman

You've already managed to compile KFX files from other sources, so I presume it's doable to reverse the process and decompile DRM-free KFX. But laborious.

Yes.

[odamizu]

@jhowell - Thank you so much! Very enlightening (and rather daunting). Perhaps I will invest in an older Kindle after all to increase my chances of maintaining access to KF8.

[DaleDe]

Originally Posted by odamizu

Thanks for all the speculations.

Assuming programmers make it possible for the rest of us to disinfect and convert KFX (and many, many thanks to these wonderful people), I'm curious what one gets when one "unpacks" a KFX.

@jhowell kindly listed the files that make up a KFX with a quick description of what they contain. How close is this to EPUB? E.g., if someone familiar with EPUB were to gain access to the main container, would they find something recognizable and editable? What's in the resource container and metadata container?



I'm also concerned by @kovidgoyal's assessment a few months ago and wondering if there's anything new to suggest the future is not so bleak:

I believe they will be like JSON and look similar to AZK both in our wiki.

Dale

[jhowell]

It's Friday night and maybe I've have a glass of wine too many, but I feel a need to comment on the current situation with Amazon's KFX. Right now it seems to be their main format for delivery of ebooks and as far as I can tell I am the only person interested enough in it to look into the details of how it works. One old retired programmer. That seems so strange to me. When I first joined MobileRead there were a lot of people delving into file formats and DRM and figuring out the whole ebook phenomenon. Now most of the developers have moved on to other things. I guess it just isn't as exciting as it once was. Such is life.

[odamizu]

Originally Posted by DaleDe

I believe they will be like JSON and look similar to AZK both in our wiki.

Thank you! This looks like more than my simple html/css skills can handle :-(

Originally Posted by jhowell

It's Friday night and maybe I've have a glass of wine too many, but I feel a need to comment on the current situation with Amazon's KFX. Right now it seems to be their main format for delivery of ebooks and as far as I can tell I am the only person interested enough in it to look into the details of how it works. One old retired programmer. That seems so strange to me. When I first joined MobileRead there were a lot of people delving into file formats and DRM and figuring out the whole ebook phenomenon. Now most of the developers have moved on to other things. I guess it just isn't as exciting as it once was. Such is life.

This makes me so very sad. Looks like I better start looking for an older Kindle and pray that Kobo (my fallback position) doesn't flip the switch on hardened Adobe DRM.

[KevinH]

FWIW, there are others that care and when they have time they will look at the format. I will eventually get the extra time away from Sigil to work on KindleUnpack again and spend some time looking at what you have done especially on how you approached converting kfx (no drm) back to its closest epub representation that will allow conversion to other formats. If the format can not be converted to a decent epub, I am simply not interested.

That said, working on the DRM version will typically require access to a PC with the proper tools and debuggers to see what is really going on. The recent change to support this format with KPC 1.19 should actually help in that regards.

The key is that there are currently ways around being forced to use that format that will actually result in a higher quality epub representation than trying to de-kfx back to epub (even if its drm could be removed). So there is not much interest by many until it is actually required.

Hope this explains things from one old programmer to another! Hope the wine was good.

KevinH

Originally Posted by jhowell

It's Friday night and maybe I've have a glass of wine too many, but I feel a need to comment on the current situation with Amazon's KFX. Right now it seems to be their main format for delivery of ebooks and as far as I can tell I am the only person interested enough in it to look into the details of how it works. One old retired programmer. That seems so strange to me. When I first joined MobileRead there were a lot of people delving into file formats and DRM and figuring out the whole ebook phenomenon. Now most of the developers have moved on to other things. I guess it just isn't as exciting as it once was. Such is life.