KFX Format

[mattmc]

Originally Posted by kovidgoyal

@odamizu: The algorithm used for encryption doesn't matter. Since other experiments have shown that the same encrypted KFX file collection works on multiple devices, that implies that whatever the algorithm, it is based on a key that is not device/account specific. So what is needed is to figure out what that key is and extract it. Once that is done, all KFX files will be decryptable trivially (that is assuming there is not another layer of separate device/account specific DRM underneath).

The problem is more a human one. The advantages that KFX brings are so minor and individual device specific that I doubt anyone is likely to spend the time to break the obfuscation. I know I for one am not motivated to do so.

Agreed, more or less. The key is obviously present somewhere. So yes, it becomes a problem of who is willing to delve deep enough.

The strings in the KFX file that state the encryption algorithm and compression method are some pretty good hints, but the key could be anywhere. Given the mention of "vouchers", and the voucher file in the KFX metadata, that would be a good place to start. But the key could also be hard-coded into the reader software (as a default for "non-DRMed" books), or the software could use a key-derivation algorithm, using something in the file or metadata as input. The possibilities are limited only by how creative Amazon wanted to be.

So while someone could be motivated enough to crack KFX, I doubt they'd do it without prying apart the reader software itself to see what it was doing.

Originally Posted by kovidgoyal

From what I have been able to deduce, KFX contains images and probably text that is pre-processed/changed to work for the particular device the KFX is sent to. As such it is fairly pointless as an ebook format, in that it is not suitable a source for conversion to another ebook format. About the only thing one would gain by reversing this format is the ability to generate your own KFX files to send to your particular Kindle.

EDIT: And I should say that in the long term this is likely to be a dead-end format anyway. Since the capabilities it brings are perfectly possible by just improving the text rendering engines for normal epub/azw3 files.

What have you seen that differed between the same KFX, delivered to different devices? I don't doubt you, I'm just hungry for details.

I can see how, if that's true, then from the Calibre viewpoint KFX is not something you'd invest time in.

Originally Posted by odamizu

kovidgoyal and mattmc for your patient and enlightening comments. (In my next life, I want to be software developer!)

No problem

Originally Posted by odamizu

I'm not so interested KFX's enhanced typography. My primary concern is maintaining the ability to tweak ebooks and convert to different formats if Amazon phases out MOBI and AZW3, leaving KFX as their only format. Hopefully that won't happen any time soon, especially given kovidgoyal's comment about KFX being an unsuitable source for conversion. Yikes.

They seem to be heading that way...

[JSWolf]

Originally Posted by mattmc

They seem to be heading that way...

To phase out Mobi & KF8, Amazon would have to abandon every 1st gen, 2nd gen, K3, KK, Kindle Touch, Kindle Basic 1, Kindle Basic 2, DX, DXG, older versions of Kindle programs, and any other hardware I left out.

Do you really think Amazon would abandon all of these users for KFX? I would say Amazon would not do that.

[Purple Lady]

Originally Posted by jhowell

I am more concerned about Amazon (and their lawyers) than I am about MobileRead. I feel that I have avoided crossing any lines with kfxmeta since it only extracts information that is plainly visible in the metadata.kfx file associated with each KFX book. Amazon has taken no steps to hide this information.

I have intentionally stayed away from examining the details of encrypted KFX files. As can be seen in the first post in this thread, even "Simultaneous Device Usage: Unlimited" books contain strings like "amzn1.drm-key.v1" and I don't see Amazon calling these books "DRM-Free" anywhere. I don't want to be put in a position of having to defend against an accusation of DMCA violation so I am going to avoid reverse engineering this aspect of KFX. (But I will be happy if someone else does so.)

Tor Books used to say in the description that the book was DRM free as requested by the publisher. I looked at a few books from Tor and that phrase is no longer in the description, but it is inside the book. If the book says inside that it is DRM free, I don't think it would be a DMCA violation.

[kovidgoyal]

@mattmc: My deduction that KFX is device specific is simply from data in this thread. Namely:

1) Images are grayscaled for e-ink screens

2) Images are re-compressed using a higher compression ratio (JPEG XR)

3) Since amazon has only enabled "advanced" typography for this format, it stands to reason that there is some pre-processing done to make those features easier to implement. For instance, adding soft-hyphens to the text, to avoid the layout engine on the Kindle from having to do that at runtime.
It is unlikely in my opinion that the HTML/CSS is simply copied unchanged from azw3 to kfx. Of course this cannot be known for certain unless the obfuscation is broken.

[jhowell]

Originally Posted by mattmc

Interesting. So, because all of the files are "DRMed", even if they are not locked to a particular user or device, it would be a violation of...copyright?...to crack them open? I admit I'm not familiar with DMCA and how it applies to DRM and so on. I thought the reason why de-DRMing software can legally exist is because it is "pitched" as being for users who want to convert their purchased book into a different format, to read on another device or something. Like the way BitTorrent is for "sharing files", even if it gets used for piracy.

It may not be a copyright violation to break DRM for public-domain books, but the DMCA is more broad than that.

Originally Posted by Purple Lady

Tor Books used to say in the description that the book was DRM free as requested by the publisher. I looked at a few books from Tor and that phrase is no longer in the description, but it is inside the book. If the book says inside that it is DRM free, I don't think it would be a DMCA violation.

Those books are still available to you without DRM, just not in KFX format.

It appears that protection is being applied to all KFX books. The first three bytes in the main file of every KFX book I have checked are 'D', 'R', 'M'. I wouldn't want to try to defend the position that it isn't actual DRM in court.



I have heard enough about the DMCA to be cautious. Some relevant quotes from Wikipedia (highlights mine):

It criminalizes production and dissemination of technology, devices, or services intended to circumvent measures (commonly known as digital rights management or DRM) that control access to copyrighted works. It also criminalizes the act of circumventing an access control, whether or not there is actual infringement of copyright itself.

The second portion (17 U.S.C. 1201) is often known as the DMCA anti-circumvention provisions. ... The section does not include a fair use exemption from criminality nor a scienter requirement, so criminal liability could attach even unintended circumvention for legitimate purposes.

The DMCA has had an impact on the worldwide cryptography research community, since an argument can be made that any cryptanalytic research violates, or might violate, the DMCA. The arrest of Russian programmer Dmitry Sklyarov in 2001, for alleged infringement of the DMCA, was a highly publicized example of the law's use to prevent or penalize development of anti-DRM measures. While working for ElcomSoft in Russia, he developed The Advanced eBook Processor, a software application allowing users to strip usage restriction information from restricted e-books, an activity legal in both Russia and the United States. Paradoxically, under the DMCA, it is not legal in the United States to provide such a tool. Sklyarov was arrested in the United States after presenting a speech at DEF CON and subsequently spent nearly a month in jail. The DMCA has also been cited as chilling to legitimate users, such as students of cryptanalysis (including, in a well-known instance, Professor Edward Felten and students at Princeton), and security consultants such as Niels Ferguson, who has declined to publish information about vulnerabilities he discovered in an Intel secure-computing scheme because of his concern about being arrested under the DMCA when he travels to the U.S.

There may be a reason why KFX is currently only available on the most locked down platforms. The DMCA also applies to Amazon's copyrighted software. Delving into how KFX works by examining software could be considered a criminal activity since Amazon has taken steps to protect access to that software. See this article on the same issue related to Volkswagen's misdoings.

This code was shielded from watchdogs’ investigation by the anti-circumvention provision of the DMCA.

Others may do as they wish. I am going to be cautious.

[jhowell]

Originally Posted by mattmc

The key is obviously present somewhere. So yes, it becomes a problem of who is willing to delve deep enough.

If anyone has already cracked KFX DRM they may have good reasons to keep it to themselves. There are the DMCA issues that I have already raised.

Beside that, it is still early days for KFX. It is only supported by a handful of devices. It would still be easy for Amazon to further enhance their DRM. The apparent lack of locking to a particular device/account is a good thing for users. It would be bad to give Amazon incentive to change that. Over time it will become harder for them to change the DRM scheme for KFX. Best for us users if hackers keep quiet until 'tools' are truly needed.

Originally Posted by odamizu

I'm not so interested KFX's enhanced typography. My primary concern is maintaining the ability to tweak ebooks and convert to different formats if Amazon phases out MOBI and AZW3, leaving KFX as their only format. Hopefully that won't happen any time soon, especially given kovidgoyal's comment about KFX being an unsuitable source for conversion. Yikes.

Originally Posted by mattmc

They seem to be heading that way...

If Amazon blocks the availability of non-KFX formats then it will be time to start worrying. We are far from there yet and it may never happen. If it turns out that Kovid and KevinH are right and it is not a suitable source for conversion then those who care will just purchase their books elsewhere.

[jhowell]

Originally Posted by mattmc

What have you seen that differed between the same KFX, delivered to different devices? I don't doubt you, I'm just hungry for details.

I have downloaded the same book on my iPad and PW2 and there are definitely differences. Images in iPad KFX are color JPEG instead of grayscale JPEG-XR.

The main DRMION file, which apparently holds the non-image data, differs in size and (unreadable encrypted) content. Though as far as I can tell the text renders the same on both devices.

[jhowell]

One small piece of information I have found is that the metadata.kfx file is optional. I have downloaded several book samples in KFX format and none of them have had an associated metadata.kfx file.

[mattmc]

Originally Posted by JSWolf

To phase out Mobi & KF8, Amazon would have to abandon every 1st gen, 2nd gen, K3, KK, Kindle Touch, Kindle Basic 1, Kindle Basic 2, DX, DXG, older versions of Kindle programs, and any other hardware I left out.

Do you really think Amazon would abandon all of these users for KFX? I would say Amazon would not do that.

I don't think that Amazon is likely to "abandon" users, but that's not what I was saying. In the past, Amazon issued firmware updates to everything but (correct me if I'm wrong) the K1, K2, and DX, to make them able to read KF8. I don't know what's stopping them from doing that again, but for KFX. An almost-all-KFX scene could be right around the corner, as fast as Amazon's firmware teams can work.

Originally Posted by jhowell

I have downloaded the same book on my iPad and PW2 and there are definitely differences. Images in iPad KFX are color JPEG instead of grayscale JPEG-XR.

The main DRMION file, which apparently holds the non-image data, differs in size and (unreadable encrypted) content. Though as far as I can tell the text renders the same on both devices.

Originally Posted by kovidgoyal

@mattmc: My deduction that KFX is device specific is simply from data in this thread. Namely:

1) Images are grayscaled for e-ink screens

2) Images are re-compressed using a higher compression ratio (JPEG XR)

3) Since amazon has only enabled "advanced" typography for this format, it stands to reason that there is some pre-processing done to make those features easier to implement. For instance, adding soft-hyphens to the text, to avoid the layout engine on the Kindle from having to do that at runtime.
It is unlikely in my opinion that the HTML/CSS is simply copied unchanged from azw3 to kfx. Of course this cannot be known for certain unless the obfuscation is broken.

Interesting, okay then. I suppose this is only in keeping with Amazon's past patterns. Kindlegen creates a MOBI and KF8, and then the appropriate version of the file is stripped out and delivered, based on the consumer's device. (KCR is also created, presumably, for Kindle Cloud Reader and K4iOS.) So we're looking at something similar, but taken to the next level.

@kovid, On the soft hyphens, maybe you can clear something up for me. What benefit would there be to adding those to the file itself? What if the user sets a font size that doesn't give a word a reason to break on a particular line? Then you just have a hyphen in the middle of a word...unless the rendering engine is hiding them when they're not necessary, or something, and then you haven't gained anything. I suppose they could be inserting invisible markers in the middle of words that the rendering engine uses as clues to do soft-hyphenation.

Originally Posted by jhowell

It appears that protection is being applied to all KFX books. The first three bytes in the main file of every KFX book I have checked are 'D', 'R', 'M'. I wouldn't want to try to defend the position that it isn't actual DRM in court. ...

I have heard enough about the DMCA to be cautious. Some relevant quotes from Wikipedia (highlights mine): ...

Others may do as they wish. I am going to be cautious.

Thanks for those quotes, I feel sufficiently warned. For now, I'm just going to hope that Amazon gives eBook creators the tools we need to proof these things. If the entire Kindle ecosystem turns to KFX and there's nothing forthcoming from Amazon, then that bridge will have to be crossed.

Originally Posted by Hitch

Just getting caught up with this thread, which thus far, has been pretty darned amusing. ...

I'm looking forward to the enhanced capabilities. I'm dead tired of the idea that a Fiverr can do what we do (or the ubiquitous "upload your Word file and we'll automagically convert it to..."). It's annoying as s**t.

So glad to have entertained you, Hitch

I'm curious what your take is on the proofing aspect, since you probably have the most skin in the game out of anybody, and perhaps the strongest working relationship with Amazon. Have you spoken with anyone there about giving creators the ability to proof stuff in KFX? Is it too early to get into that yet? Or are you confident enough in the simplicity of your files that you're not overly concerned about the KFX conversion playing havoc with them?

Would appreciate your 2 cents

(Also, how does KFX raise the bar of required skill to create an eBook? I'm not clear on that point.)

[DaleDe]

Originally Posted by jhowell

If anyone has already cracked KFX DRM they may have good reasons to keep it to themselves. There are the DMCA issues that I have already raised.

Beside that, it is still early days for KFX. It is only supported by a handful of devices. It would still be easy for Amazon to further enhance their DRM. The apparent lack of locking to a particular device/account is a good thing for users. It would be bad to give Amazon incentive to change that. Over time it will become harder for them to change the DRM scheme for KFX. Best for us users if hackers keep quiet until 'tools' are truly needed.

If Amazon blocks the availability of non-KFX formats then it will be time to start worrying. We are far from there yet and it may never happen. If it turns out that Kovid and KevinH are right and it is not a suitable source for conversion then those who care will just purchase their books elsewhere.

It may be by design that Amazon doesn't want this to be a suitable conversion source. Since they are the only ones that have it they are the only ones that benefit from the improvements. Note that MOBI was originally exactly the same. The format has never been public nor is it likely that it will ever be. There are many sources outside the control of DCMA.

I am also beginning to believe that AZK is an intermediate design and will be dropped in favor of KFX, particularly due to the similarities and the fact that KFX is now on Apple based products.

Dale

[kovidgoyal]

@mattmc: Inserting soft hyphens before hand means the text rendering engine does not have to figure out the positions at which it is suitable to insert hyphens in a word by itself. This makes its performance somewhat better and means the rendering engine does not have to ship with large collections of language specific hyphenation rules.

[jhowell]

Originally Posted by mattmc

I don't think that Amazon is likely to "abandon" users, but that's not what I was saying. In the past, Amazon issued firmware updates to everything but (correct me if I'm wrong) the K1, K2, and DX, to make them able to read KF8. I don't know what's stopping them from doing that again, but for KFX. An almost-all-KFX scene could be right around the corner, as fast as Amazon's firmware teams can work.

This seems very unlikely to me. Even if at some point they switch over to KFX for new books, I expect that they will retain the existing older formats for current books. (Also, there are already books that work on some devices and not others. KFX-only books would just be an extension of that.)

I highly doubt they would put any resources into firmware upgrades for outdated models. If anything, when the time comes, they could offer an attractive trade-in or sale on the lowest-end current model to promote upgrades.

Originally Posted by mattmc

@kovid, On the soft hyphens, maybe you can clear something up for me. What benefit would there be to adding those to the file itself? What if the user sets a font size that doesn't give a word a reason to break on a particular line? Then you just have a hyphen in the middle of a word...unless the rendering engine is hiding them when they're not necessary, or something, and then you haven't gained anything. I suppose they could be inserting invisible markers in the middle of words that the rendering engine uses as clues to do soft-hyphenation.

See the Wikipedia page for more information on soft hyphens:

In this context, the soft hyphen may also be called a discretionary hyphen or optional hyphen. It serves as an invisible marker used to specify a place in text where a hyphenated break is allowed without forcing a line break in an inconvenient place if the text is re-flowed. It becomes visible only after word wrapping at the end of a line. The soft hyphen's Unicode semantics and HTML implementation are in many ways similar to Unicode's zero-width space.

[Hitch]

Originally Posted by mattmc

I don't think that Amazon is likely to "abandon" users, but that's not what I was saying. In the past, Amazon issued firmware updates to everything but (correct me if I'm wrong) the K1, K2, and DX, to make them able to read KF8. I don't know what's stopping them from doing that again, but for KFX. An almost-all-KFX scene could be right around the corner, as fast as Amazon's firmware teams can work.

Dunno. They've never gone the route of Apple, and I hope that they don't. I am STILL seeing lotsa KF7 devices in my daily trade.

Interesting, okay then. I suppose this is only in keeping with Amazon's past patterns. Kindlegen creates a MOBI and KF8, and then the appropriate version of the file is stripped out and delivered, based on the consumer's device. (KCR is also created, presumably, for Kindle Cloud Reader and K4iOS.) So we're looking at something similar, but taken to the next level.

Presumably. I haven't delved into this, as you lads (and ladies, if anyone else is here?) have.

@kovid, On the soft hyphens, maybe you can clear something up for me. What benefit would there be to adding those to the file itself? What if the user sets a font size that doesn't give a word a reason to break on a particular line? Then you just have a hyphen in the middle of a word...unless the rendering engine is hiding them when they're not necessary, or something, and then you haven't gained anything. I suppose they could be inserting invisible markers in the middle of words that the rendering engine uses as clues to do soft-hyphenation.

I don't see the advantage, either, but....I am not Amazon. You just never know what on earth they're going to do. Some of their rendering choices have left me speechless.

Thanks for those quotes, I feel sufficiently warned. For now, I'm just going to hope that Amazon gives eBook creators the tools we need to proof these things. If the entire Kindle ecosystem turns to KFX and there's nothing forthcoming from Amazon, then that bridge will have to be crossed.



So glad to have entertained you, Hitch

Oh, anytime. I generally begin, lately, to feel like Banquo's Ghost, here at MR. I find my name in the oddest places. It's a bit weird.

I'm curious what your take is on the proofing aspect, since you probably have the most skin in the game out of anybody, and perhaps the strongest working relationship with Amazon. Have you spoken with anyone there about giving creators the ability to proof stuff in KFX? Is it too early to get into that yet? Or are you confident enough in the simplicity of your files that you're not overly concerned about the KFX conversion playing havoc with them?

Would appreciate your 2 cents

(Also, how does KFX raise the bar of required skill to create an eBook? I'm not clear on that point.)

I have NOT yet discussed KFX with Amazon. I don't know enough about it to discuss it with them intelligently, and as youse guys know, I'm not a programmer. It doesn't mean I'm not interested in this stuff--I am, ridiculously--but my more-than-fulltime job keeps me a bit too busy to be able to take the time to discover what you are working on. I wish I could, seriously.

Or are you confident enough in the simplicity of your files that you're not overly concerned about the KFX conversion playing havoc with them?

Oh...would I that that were true. Oh, no, not at ALL. Honestly, if they're heading into "advanced typography," it worries me, as their previous experiments, surrounding fonts especially, have been fairly disastrous for those of us with aforementioned skin in the game. Only those formatters who do things like "upload Word files," etc., are probably safe--in terms of "advanced typography," I mean--around this. (n.b.: this assumes that they didn't do OTHER stupid things...)

We've tried to be careful around using fonts, but as I've mentioned before, we do a LOT of white-label work, in which we essentially have no say, other than "how high?"

In terms of your question about "Also, how does KFX raise the bar of required skill to create an eBook?" my answer is a) I don't know, and b) I suspect it might.

To wit: here's what we all don't know: we all don't yet know if every bloody book out there, that has escaped into the KDP, over the last 7-ish years, will be reprocessed.

Now, having been the victim, more than once, of changes at the KDP, in which X worked on Monday, and didn't on Friday, that part scares my skivvies.

We don't know if "advanced typography" means that EVERY SINGLE UPLOADED WORD FILE out there with empty paras in it is going to be reprocessed, with results meaning NO blank space (top margin, bottom margin, whatever). We all know that we've seen kluges over the years to do things (thank God, I think I only did this ONCE, in 2009-10), like center a title page, vertically. Using the dreaded empty paras, OR, for that matter, who amongst us--now, fess up, boys--hasn't used an NBSP in a table, to make the content "look right?"

So: if all the books that e'er were, and e'er shall be, are going to be REPROCESSED, I foresee...bad. We already get a ridiculous number of requests to "fix my book" (all for $5, or $25, or whatever), from users of this or that program, or software, or browser-based SAS app. (No, Kovid, I'm not talking about Calibre, particularly.)

Can't know yet. We're going to have to wait, to see what happens. I'm holding my breath. I already get people who come to me (no, I'm not making this up) 3, 4 YEARS after we've made an ePUB for them, upset because NOW, some 4 years later, the damned ePUB won't intake at iBooks or wherever, because the bloody thing won't validate, naturally. And that's the wee percentage that venture forth from Amazon, into the scary waters of "other retailers." If all the books are RE-processed, and if what you've all seen, thus far, holds....And not even just that. Amazon's made a boatload of OTHER changes over the years, but they didn't mass-re-process the books. They made the changes at the intake, or the PW (Publishing Workflow), e.g., the SRL change. When that happened, books that had been made earlier, say, 2010, would now not work correctly, for SRL, etc., because they'd been made under an earlier standard/requirement/PW. Who knows what on earth will happen if, essentially, all the MOBIs are effectively put through the PW again? Zoiks.

(Sorry, Kovid: but he asked. I know you wanted to keep this on the tech, not typographic side, but it would be rude of me not to reply.)

Hitch

[badgoodDeb]

(I'm fascinated, Hitch!)

[Hitch]

Originally Posted by badgoodDeb

(I'm fascinated, Hitch!)

TY, Badgood. Me, too.

Hitch