KFX Format

[AaronShep]

Originally Posted by JSWolf

Please delete any text relating to turning off hyphenation in Amazon eBooks. This just leads to abuse. I've seen many ePub with CSS commands to turn off hyphenation for ADE and webkit and that's just wrong on so many levels.

And yet Amazon KDP itself suggested to me that there were valid uses for that.

Do you really think it's appropriate, for instance, to hyphenate a heading? Can you show me a single print book that allows it?

Aaron

[JSWolf]

Originally Posted by AaronShep

And yet Amazon KDP itself suggested to me that there were valid uses for that.

Do you really think it's appropriate, for instance, to hyphenate a heading? Can you show me a single print book that allows it?

Aaron

But some who make eBooks just decide to remove hyphens everywhere. It's too easy to be abused when used.

[AaronShep]

Originally Posted by DuckieTigger

Maybe you could be nice enough to give an example of program that outputs a nice typeset book no matter how long it takes.

If you go into the Compatibility options of Microsoft Word, you'll see one for enabling "WordPerfect-style" justification. This style allows both expansion and compression of spaces between words, instead of just expansion, as you would normally see in a word processor or Web browser. And this style of justification, with both compression and expansion, is what you'll find in any page layout program. Without it, spacing between words in justified text is generally excessive, especially in narrow columns.

This option has been in Word for at least two decades, and as the name suggests, the style was standard in WordPerfect even before that. So, I doubt it would unduly tax even the processors in today's Kindles. In fact, it would take less processing power than the kerning introduced with the new layout engine.

The reason you don't get this on Kindle is that the Kindle developers themselves know nothing about typography. The advances announced for the new layout engine are just features of Webkit that the Kindle team has finally decided to turn on. They're just throwing switches. If book-style typography was available in Webkit, we'd get it on Kindle. But as it is, Amazon continues to give us screenshots of poor typography and bragging about its advances.

Aaron

[AaronShep]

Originally Posted by JSWolf

But some who make eBooks just decide to remove hyphens everywhere. It's too easy to be abused when used.

I could say the same about the English language -- especially on Kindle.

Aaron

[DuckieTigger]

@Aaron: that is all you want? Expansion and compression for justification and you are all happy? How about just embedding a font with really really narrow space character and call it done. The Kindle will leave the narrow space alone until it decides to stretch it out.

For a second I thought you meant aesthetically pleasing hyphenation, optically appearing justified text (not simply forced into a rectangle), ...

[AaronShep]

Originally Posted by DuckieTigger

@Aaron: that is all you want? Expansion and compression for justification and you are all happy? How about just embedding a font with really really narrow space character and call it done. The Kindle will leave the narrow space alone until it decides to stretch it out.

For a second I thought you meant aesthetically pleasing hyphenation, optically appearing justified text (not simply forced into a rectangle), ...

That was a prime example. No, I'm not asking for anything fancy, just elementary book typography with freedom from ill-conceived overrides. In other words, what we're not getting from the Kindle.

I would also like not to have added space between indented paragraphs. Respect for nonbreaking spaces. I can dream about nonbreaking hyphens too. I discuss some of these on my blog, and more in my books.

By the way, I just posted on my blog about problems related to JPEG XR, the new graphics format for KFX.

http://www.newselfpublishing.com/blog/

Aaron

[kovidgoyal]

Guys, can we please restrict this thread to the KFX format, I use it to keep in touch with whatever new data we learn about KFX, since I dont have a KFX enabled device myself -- unrelated discussion on typography makes that harder. There are plenty of other forums this discussion can be moved to.

[odamizu]

I'm the first to admit I know next to nothing about this, but given the findings by james194zt with regard to AES/CBC/PKCS5 encryption with SHA2 256Kb hashes, is cracking KFX even possible at this time? Or is it a waiting game until Amazon releases (if they release) a KFX-capable Kindle for PC/Mac app or kfxgen utility?

(I'm not being snarky or sarcastic. I honestly want to understand the process as much as my limited brain can grasp.)

Also, I'm really worried mattmc is right when he says:

Originally Posted by mattmc

I think the easiest answer to "why KFX" is probably the opacity. If the whole file is encrypted in AES/CBC, regardless of whether the file is DRM'd or not...I mean, it's pretty obvious. They don't want anyone looking at the internals. But the "enhanced typography" certainly provides a good cover.

I mean, I could be wrong, for sure. If Amazon starts passing out the kfxgen utility so creators can properly proof their books, then I was wrong.

[DiapDealer]

@Kovid: Agreed.

I'm as guilty as any for the latest derailment, but we need to reserve this space for discussion about the technical specifications concerning the KFX format. The guts, so to speak. Discussions about things we like/don't like about it, or what happens to the formatting of books that get converted to it, and general typography disputes need to be taken elsewhere.

This is a "what makes it tick" thread. As such, "how its contents get rendered" is irrelevant right now.

[mattmc]

Originally Posted by jhowell

I have intentionally stayed away from examining the details of encrypted KFX files. As can be seen in the first post in this thread, even "Simultaneous Device Usage: Unlimited" books contain strings like "amzn1.drm-key.v1" and I don't see Amazon calling these books "DRM-Free" anywhere. I don't want to be put in a position of having to defend against an accusation of DMCA violation so I am going to avoid reverse engineering this aspect of KFX. (But I will be happy if someone else does so.)

Interesting. So, because all of the files are "DRMed", even if they are not locked to a particular user or device, it would be a violation of...copyright?...to crack them open? I admit I'm not familiar with DMCA and how it applies to DRM and so on. I thought the reason why de-DRMing software can legally exist is because it is "pitched" as being for users who want to convert their purchased book into a different format, to read on another device or something. Like the way BitTorrent is for "sharing files", even if it gets used for piracy.

Anyway, you make a good point; I don't particularly want to be targeted by Amazon.

Originally Posted by kovidgoyal

Guys, can we please restrict this thread to the KFX format...

Originally Posted by DiapDealer

...we need to reserve this space for discussion about the technical specifications concerning the KFX format.

Agreed. Although, I do think that @AaronShep's data on the new JPEG XR image format is relevant.

Originally Posted by odamizu

I'm the first to admit I know next to nothing about this, but given the findings by james194zt with regard to AES/CBC/PKCS5 encryption with SHA2 256Kb hashes, is cracking KFX even possible at this time? Or is it a waiting game until Amazon releases (if they release) a KFX-capable Kindle for PC/Mac app or kfxgen utility?

In my estimation, yes, we wait for one of those two outcomes. Obviously it would be best if Amazon just gave us the kfxgen utility. That would solve my particular problem of proofing.

A thought did occur to me: since Kindles are basically lightweight Linux machines, could you SSH into one from your desktop and attach GDB to whatever process is reading the files? Kinda a question for the crazy hackers over in the Kindle Developers' Corner.

[Hitch]

Originally Posted by DiapDealer

I don't see the non-breaking space issue as a tragedy, myself. I've always felt that ebook makers who use paragraphs with only non-breaking spaces (or a non-breaking space followed by a br tag) for vertical spacing were shooting themselves in the foot all along. This change just confirms that.

Besides, they're (Amazon) not the first to have renderers that ignored empty paragraphs (or paragraphs with only an nbsp). That's why the nbsp followed by a br tag convention came into play in the first place (and so long ago).

They never had a format that rendered identically across all their devices and apps anyway. Publishers, creators and selfpubbers will survive this change too. They always do. It's always involved a lot of work planning and implementing a coding strategy to achieve similar rendering on the widest possible range of devices and apps (and formats) without resorting to maintaining 14 different versions of your ebook. This doesn't really change that.

And if it forces people to start using css for vertical spacing (like they should have been doing all along), I'm all for it. Makes it exponentially easier for the person [Hi Hitch!] that's eventually going to get called in to fix it all, at some point down the road, anyway!

Hi, DD:

Just getting caught up with this thread, which thus far, has been pretty darned amusing. Yeah, verily, a long time ago and some galaxies away--in my MBPC days, in the aughts--I may have used the "empty para" trick--but if a bookmaker tries that with me now, they're not working for me any longer. For all the right AND wrong reasons, but mostly because, it's simply unreliable and it's bloody LAZY.

I'm looking forward to the enhanced capabilities. I'm dead tired of the idea that a Fiverr can do what we do (or the ubiquitous "upload your Word file and we'll automagically convert it to..."). It's annoying as s**t.

Hitch

[kovidgoyal]

@odamizu: The algorithm used for encryption doesn't matter. Since other experiments have shown that the same encrypted KFX file collection works on multiple devices, that implies that whatever the algorithm, it is based on a key that is not device/account specific. So what is needed is to figure out what that key is and extract it. Once that is done, all KFX files will be decryptable trivially (that is assuming there is not another layer of separate device/account specific DRM underneath).

The problem is more a human one. The advantages that KFX brings are so minor and individual device specific that I doubt anyone is likely to spend the time to break the obfuscation. I know I for one am not motivated to do so.

From what I have been able to deduce, KFX contains images and probably text that is pre-processed/changed to work for the particular device the KFX is sent to. As such it is fairly pointless as an ebook format, in that it is not suitable a source for conversion to another ebook format. About the only thing one would gain by reversing this format is the ability to generate your own KFX files to send to your particular Kindle.

EDIT: And I should say that in the long term this is likely to be a dead-end format anyway. Since the capabilities it brings are perfectly possible by just improving the text rendering engines for normal epub/azw3 files.

[Little.Egret]

Originally Posted by AaronShep

And yet Amazon KDP itself suggested to me that there were valid uses for that.

Do you really think it's appropriate, for instance, to hyphenate a heading? Can you show me a single print book that allows it?

Aaron

Depends what you mean by 'heading'

Consider the Victorian-style chapter headings "in which our hero ..."

If these use several lines the book designer will probably choose to hyphenate.

[odamizu]

kovidgoyal and mattmc for your patient and enlightening comments. (In my next life, I want to be software developer!)

I'm not so interested KFX's enhanced typography. My primary concern is maintaining the ability to tweak ebooks and convert to different formats if Amazon phases out MOBI and AZW3, leaving KFX as their only format. Hopefully that won't happen any time soon, especially given kovidgoyal's comment about KFX being an unsuitable source for conversion. Yikes.

[jhowell]

Originally Posted by AaronShep

You would automatically get KFX if the particular publisher's version you purchased had been converted. But you would NOT get KFX if Amazon had only converted publisher versions submitted AFTER your purchase. For those, you would have to contact Amazon and ask them to replace your older version with the new one.

Originally Posted by jhowell

I believe that you have found the problem. I must have an old revision of the book associated with my account. ... I will see if I can get Amazon customer service to update that book under my account.

Just to follow up on this. I called Kindle Support and they were able to push a book update that allowed the book that previously would only download as AZW3 to be delivered as KFX.