KFX Format

[Hitch]

Originally Posted by AaronShep

I think Amazon's statements have made it very clear that they're gradually working their way through at the least the majority of all Kindle books, and probably all. That's the only reason it's taking so long. They've also stated that books may be reprocessed more than once and whenever necessary, as features change. I imagine that slower sellers get low priority, but that would only be temporary protection.

Aaron

Yes. Even ignoring Amazon's comments about it, the tests conducted within this thread alone indicate that reprocessing is definitely occurring. My ponderment is...I wonder when the authors/self-pubs, etc. will figure out if their book was improved thereby, or demolished thereby?

Hitch

[jhowell]

I decided to do a little digging into the release history of KFX. I know that the Kindle iOS app has supported it since version 4.9 (May 2015) and recent e-ink Kindles support it as of version 5.6.5 (August 2015).

I was wondering when support was added to Fire devices so I did a little snooping. By checking downloadable firmware updates for strings related to KFX I was able to see that support was first added in the 4th quarter of 2013 with the release of the 3rd Generation Fire devices. This is much earlier than I expected!

It seems that the roll out of KFX has been going on for a lot longer and at a much slower pace than I thought.


----Edit----
Here is a bit more information on my method:

Since Amazon usually only lists the most recent version of each device's firmware on their website, I searched in Google for older firmware versions. I was able to find quite a few. (I wouldn't recommend actually installing anything found that way; too much chance of tampering.)

The Fire firmware files are archives in zip format. I expanded them and then searched for strings associated with the KFX format, such as 'yj' and 'kfx'. After weeding through many false positives I was able to come up with longer strings that work better, including 'yjsdk' and 'parseKFXContainer'. I found that when these strings occur they are most often in the file named '/lib/libKRF.so'.

Since the file dates within the archives are unreliable (most are 9/6/2011), I used the 'post-timestamp' (in Unix format) found in /META-INF/com/android/metadata to determine when each release was actually created.

I wasn't able to find every release, but I found enough to narrow the window of when KFX first appeared to late 2013. The earliest release I found with KFX support was Kindle Fire HDX 8.9 (3rd Generation) OS version 3.0.5, dated 10/29/2013. The second generation Fires also received an update with KFX around the same time.

[KevinH]

Hi,

Forgive my ignorance but what processor is used on the Fire? Is it x86 family or ARM or ?

Thanks,

Kevin

[jhowell]

Originally Posted by KevinH

Forgive my ignorance but what processor is used on the Fire? Is it x86 family or ARM or ?

They are ARM processors running a customized Android OS.

The early appearance of KFX on Fire tablets makes me believe that it was the platform used for early testing. If Amazon is continuing to use Fire in this way then perhaps there are hints at more upcoming Kindle features hidden in the new Fire tablet firmware.

[jhowell]

KevinH passed on some information that he discovered on-line. In order to avoid possible trouble for the source of this information I won't be too specific. The main points that I picked out are:

• Yellow Jersey was a project to bring interactive magazines to Fire tablets.
• It included PDF to KFX conversion.
• KFX does stand for Kindle Format 10.

This information explains why there are references to 'yj' and 'kfx' in Fire OS code going back to late 2013. It wasn't, as I had guessed, an early test of KFX for e-books, but actually the roll-out of magazine capability.

Perhaps the re-purposing of KFX as an e-book format was a later idea and not the original plan. Re-purposing existing software is much easier and less expensive than developing something new from scratch. Some the the quirkiness of KFX and its limited availability across Amazon's e-book collection may be due to limitations in the format carried over from the original project.

The mention of PDF as a source is a bit disturbing. The hidden contents of KFX is probably very unlike previous e-book formats.

[DaleDe]

Originally Posted by jhowell

KevinH passed on some information that he discovered on-line. In order to avoid possible trouble for the source of this information I won't be too specific. The main points that I picked out are:

• Yellow Jersey was a project to bring interactive magazines to Fire tablets.
• It included PDF to KFX conversion.
• KFX does stand for Kindle Format 10.

*stuff deleted**

The mention of PDF as a source is a bit disturbing. The hidden contents of KFX is probably very unlike previous e-book formats.

I would expect this and don't find it disturbing. Magazines are typically released in PDF format by the publisher if they have a eBook version at all. They are similar to text books which get the same treatment from publishers. Amazon has already released a AZW4 format that is a mobi wrapper for PDF content. However it is likely inadequate for many devices. I am not surprised that this was the impetus to drive a new format.

Dale

[jhowell]

I continue to see a lot of books in my Kindle e-book purchases that do not support enhanced typesetting, even though they are regular novels with no apparent special formatting. I decided to look into this since it could provide useful information for authors/publishers wondering why certain books do not allow enhanced typesetting.

In order to research this I took from my collection a sample of 15 books that support enhanced typesetting and 20 books that do not. I then did an inventory of HTML tags and CSS properties in these books, based on the KF8/AZW3 version. Finally, I compared the two sets to see if there was anything unusual I could find in the books that do not support enhanced typesetting.

I found two sets of tags that appear in some books that do not support enhanced typesetting, but never appear in the sample of books that do support it:

• table/tr/td tags
  
• h3 and higher heading tags

Could these be e-book features not (yet) implemented in KFX?

(Due to the small sample size of this experiment the results could just have come about by chance, so take this with a grain of salt.)

[shamanNS]

I think they are just doing it randomly. Because for the same recently (06. and 08.10.2015) published book of the two different Kindle Editions (one by Tor, one by Gollanz) one has (Tor) and one has not (Gollanz) enhanced typography enabled.

[DaleDe]

Originally Posted by jhowell

I continue to see a lot of books in my Kindle e-book purchases that do not support enhanced typesetting, even though they are regular novels with no apparent special formatting. I decided to look into this since it could provide useful information for authors/publishers wondering why certain books do not allow enhanced typesetting.


I found two sets of tags that appear in some books that do not support enhanced typesetting, but never appear in the sample of books that do support it:

• table/tr/td tags
  
• h3 and higher heading tags

Could these be e-book features not (yet) implemented in KFX?

(Due to the small sample size of this experiment the results could just have come about by chance, so take this with a grain of salt.)

I can't imagine h3 and higher tags causing a problem but tables could be a good clue.

Dale

[gtoal]

Originally Posted by AaronShep

It's a nice sentiment, but what I've seen over and over is that people are conditioned to favor poor typography because that's what they're used to nowadays. So, principles of good typography that were developed over centuries have been discarded in favor of the preferences of engineers raised on Microsoft Word. I don't think anyone with a grounding in book typography would see the Kindle's typography as any better than a necessary evil.

Still, I wouldn't object to reader control if that's actually what we were offered. But if it's a choice between Amazon's dictating the typography or me, I won't willingly leave it in the hands of people who have proven themselves clueless.

Aaron

As an old TeX hand, it has appalled me for years that very few typesetting systems approach what TeX achieved thirty years ago, and those which do make an effort have done so pretty much by straight copying of TeX's core algorithms. However that gripe is not the point of my post; rather it is to ask if anyone is interested in a piece of code I wrote a few years ago when the Kindle first came out, which takes an HTML file and 'manually' inserts & shy; characters into words at the points determined by TeX's hyphenation algorithm. I never released it at the time I wrote it because the use of & shy; as the equivalent of TeX's \- discretionary hyphen was not its intended purpose; however it worked, and I used it, but I didn't want to release something that would entrench non-standard codification in published documents, in case Amazon corrected their reading software and suddenly all the soft hyphens became visible. However at this point that is unlikely to happen (in fact I think the original definition of the soft hyphen character has been forgotten - it was probably a faulty specification anyway...) and if the discretionary-hyphen-insertion software is still of interest I'll package it up and release it.

PS The original meaning of & shy; is apparently not entirely forgotten: this old posting summarizes the issues: http://www.cs.tut.fi/~jkorpela/shy.html

[kovidgoyal]

@gtoal dont want to dampen your enthusiasm, but see https://www.mobileread.com/forums/sho....php?p=2456848

[gtoal]

Originally Posted by kovidgoyal

@gtoal dont want to dampen your enthusiasm, but see https://www.mobileread.com/forums/sho....php?p=2456848

Actually I'm delighted to see it; I have quite enough projects on the go at the moment that I'm happy not to take on another support commitment :-) I just felt obliged to offer the code since it seemed like it would be a solution for some folks in this thread to give them a way to avoid switching to the new format.

[AaronShep]

Originally Posted by gtoal

However that gripe is not the point of my post; rather it is to ask if anyone is interested in a piece of code I wrote a few years ago when the Kindle first came out, which takes an HTML file and 'manually' inserts & shy; characters into words at the points determined by TeX's hyphenation algorithm.

Not all Kindles support soft hyphens. Some will display a soft hyphen as a hard hyphen, whether or not it falls at the end of the line. In other words, YOU CAN'T USE SOFT HYPHENS IN KINDLE BOOKS. Amazon is very explicit about it not being supported. The only special formatting characters supported by ALL Kindles -- or supposed to be, anyway -- are the nonbreaking space and the zero width non-joiner.

This is a common error of Kindle authors that leads to desperate forum posts and viral complaints based on nothing but not being aware of Kindle guidelines. "Where did all those extra hyphens come from?!!!!" They're soft hyphens, that's all.

[Hitch]

Originally Posted by AaronShep

Not all Kindles support soft hyphens. Some will display a soft hyphen as a hard hyphen, whether or not it falls at the end of the line. In other words, YOU CAN'T USE SOFT HYPHENS IN KINDLE BOOKS. Amazon is very explicit about it not being supported. The only special formatting characters supported by ALL Kindles -- or supposed to be, anyway -- are the nonbreaking space and the zero width non-joiner.

This is a common error of Kindle authors that leads to desperate forum posts and viral complaints based on nothing but not being aware of Kindle guidelines. "Where did all those extra hyphens come from?!!!!" They're soft hyphens, that's all.

First, thanks--you beat me to it, (there are days I want to SCREAM about the endless soft-hyphen discussions) and second--that's also caused in large part by people who don't know how to clean up Review marks. Happens near-daily.

Hitch

[DuckieTigger]

Originally Posted by Hitch

First, thanks--you beat me to it, (there are days I want to SCREAM about the endless soft-hyphen discussions) and second--that's also caused in large part by people who don't know how to clean up Review marks. Happens near-daily.

Hitch

I don't give a hoot if there is no soft-hyphens suppossed to be when creating a book. As reader I only care that I can add them and the books will be hyphenated on my Voyage. And that is what the plugin for calibre is supposed to do - not as a book-creation tool.