KFX Format

[jhowell]

Thanks for the kind words, odamizu and mattmc.

By being so secretive about the workings of KFX, Amazon has made it an interesting mystery for me. I have enjoyed delving into its secrets. For me this has been driven by academic interest, rather than any practical need.

I don't sideload books. Most of my reading material is sourced from Amazon and so already supports enhanced typesetting. The few books I purchase from other sources I send to my personal documents so that they will be synchronized across my multiple reading devices, so being able to do my own conversion to KFX for sideloading has no practical use for me.

At one time I thought that being able to convert from KFX to EPUB would be useful, but that has turned out not to be true (at least at present.) The loss of HTML internal coding during EPUB to KDF/KFX conversion makes it a less preferable format for archival or conversion when compared with KF8. Since Amazon continues to make KF8 versions of purchased books easily available it would be disadvantageous to use KFX for that purpose.

Although lacking practical utility, I will probably continue investigating KFX until I stop making progress or something else piques my interest.

Most recently I have scripted the EPUB to KDF conversion process so that it is possible to do bulk conversion. This should give me more data on the equivalence between EPUB and KDF/KFX.

If I ever get it working to my satisfaction, I plan to package the EPUB to KFX conversion process into a calibre plugin. Others may have a use for that capability.

[HarryT]

Originally Posted by jhowell

If I ever get it working to my satisfaction, I plan to package the EPUB to KFX conversion process into a calibre plugin. Others may have a use for that capability.

Thank you. That would be a useful capability.

[cerem0ny]

Originally Posted by jhowell

Thanks for the kind words, odamizu and mattmc.

By being so secretive about the workings of KFX, Amazon has made it an interesting mystery for me. I have enjoyed delving into its secrets. For me this has been driven by academic interest, rather than any practical need.

I don't sideload books. Most of my reading material is sourced from Amazon and so already supports enhanced typesetting. The few books I purchase from other sources I send to my personal documents so that they will be synchronized across my multiple reading devices, so being able to do my own conversion to KFX for sideloading has no practical use for me.

At one time I thought that being able to convert from KFX to EPUB would be useful, but that has turned out not to be true (at least at present.) The loss of HTML internal coding during EPUB to KDF/KFX conversion makes it a less preferable format for archival or conversion when compared with KF8. Since Amazon continues to make KF8 versions of purchased books easily available it would be disadvantageous to use KFX for that purpose.

Although lacking practical utility, I will probably continue investigating KFX until I stop making progress or something else piques my interest.

Most recently I have scripted the EPUB to KDF conversion process so that it is possible to do bulk conversion. This should give me more data on the equivalence between EPUB and KDF/KFX.

If I ever get it working to my satisfaction, I plan to package the EPUB to KFX conversion process into a calibre plugin. Others may have a use for that capability.

Yes! Yes! Yes!
Hurry! Hurry! Hurry!
Take all my money!

[jhowell]

I have been testing the Previewer 3 beta on a large number of EPUBs and so far the results have not been encouraging.

For EPUBs that validate with both the calibre editor and EpubCheck the failure rate for conversion with the Previewer in my tests has been about 50%. For random EPUBs from various sources that render fine but may have internal coding issues the failure rate runs about 80%. Sometimes I can determine the cause of the failure from the Previewer logs, but often the error message just indicates that the conversion software crashed.

Hopefully the final release will work more reliably, but I wouldn't count on it. This high failure rate makes it more of a toy than a practical tool for generating books with enhanced typesetting.

I am curious if anyone else has been testing the Previewer and had better results.

Originally Posted by cerem0ny

Yes! Yes! Yes!
Hurry! Hurry! Hurry!
Take all my money!

Obviously you are joking here! (If not then just how much money DO you have? )

Still, I would like to set reasonable expectations. The process of converting a book from EPUB to a KFX file that can be sideloaded will need to rely on the Previewer. It is likely to be time consuming, error prone, and fail with more books than it succeeds (see above). Even when it does work you could get a similar result with less effort using the Hyphenate This! calibre plugin.

[Rizla]

It sounds like you've had some success converting an epub to kfx. Have you made any progress converting kfx to other formats?

[jhowell]

Originally Posted by Rizla

It sounds like you've had some success converting an epub to kfx. Have you made any progress converting kfx to other formats?

The Previewer (when it works) has allowed me to compare EPUBs with the resulting equivalent KDFs. I have also been able to use these results to see how different KDF data types appear in the Previewer's book renderer. That, combined with a few instances of unencrypted KFX files obtained from Amazon, has allowed me to have a pretty good idea of how KFX works, at least for reflowable e-books.

I have written some code to convert KDF/KFX back into EPUB in order to test my assumptions. That isn't anywhere near usable as a general function, just a proof of concept. I posted an example result earlier.

I don't see conversion of KFX to EPUB as being a very useful capability. First Amazon encrypts KFX books, even those that are available DRM-free in KF8. The only books that could be converted from KFX to EPUB are those you converted from EPUB to KFX in the first place. Even if decryption eventually becomes possible, KFX is far inferior to KF8 as a source for archival or conversion. The original EPUB internal HTML semantic markup is lost in the conversion of EPUB to KFX.

For example, in KFX conversion <p>, <div>, <h1>, and <blockquote> all produce the same 'text' data type with any differences reflected in the formatting style. The same goes for <b>, <em>, <i>, <small>, <span>, <strike>, <strong>, <sub>, <sup>, etc., which all become just the equivalent css styles applied to text. There is no way to tell from the result whether the original EPUB used these different tags or just used css for formatting. Also values expressed in px or pt units tend to be converted to roughly equivalent em units in KFX.

[odamizu]

Thanks again @jhowell for sharing your findings.

I'm not all that interested in converting to KFX, either. My main concern is being able to convert from KFX to other formats IF Amazon ever switches to KFX exclusively. Hopefully that won't happen anytime soon

[Rizla]

K. Thx for your answer.

[AaronShep]

Originally Posted by jhowell

Hopefully the final release will work more reliably, but I wouldn't count on it. This high failure rate makes it more of a toy than a practical tool for generating books with enhanced typesetting.

I am curious if anyone else has been testing the Previewer and had better results.

On my Mac, I get a conversion failure notice for ALL of my HTML files, even though they're syntactically perfect and the same files were being converted to KFX by Amazon after submission. BUT -- In the process, the Previewer creates a MOBI file which it leaves accessible, and if I submit THAT to the Previewer, it goes ahead and converts it successfully.

So, the initial failure seems to be from a problem with the Previewer, not my files.

[cerem0ny]

Originally Posted by jhowell

I have been testing the Previewer 3 beta on a large number of EPUBs and so far the results have not been encouraging.

For EPUBs that validate with both the calibre editor and EpubCheck the failure rate for conversion with the Previewer in my tests has been about 50%. For random EPUBs from various sources that render fine but may have internal coding issues the failure rate runs about 80%. Sometimes I can determine the cause of the failure from the Previewer logs, but often the error message just indicates that the conversion software crashed.

Hopefully the final release will work more reliably, but I wouldn't count on it. This high failure rate makes it more of a toy than a practical tool for generating books with enhanced typesetting.

I am curious if anyone else has been testing the Previewer and had better results.




Obviously you are joking here! (If not then just how much money DO you have? )

Still, I would like to set reasonable expectations. The process of converting a book from EPUB to a KFX file that can be sideloaded will need to rely on the Previewer. It is likely to be time consuming, error prone, and fail with more books than it succeeds (see above). Even when it does work you could get a similar result with less effort using the Hyphenate This! calibre plugin.

Hehehe, not much money, but hell yeah I would pay for such a plugin... thing is, the text is noticeably easier on my eyes with enhanced typesetting, so the Hyphenate This plugin isn't to my exact standards... For now I guess I'm content just converting all my books with the previewer but I would kill for a proper plugin!

[Notjohn]

Originally Posted by AaronShep

So, the initial failure seems to be from a problem with the Previewer, not my files.

Or with the Mac?

Apple and Amazon have never played well together.

[cerem0ny]

Originally Posted by jhowell

The Previewer (when it works) has allowed me to compare EPUBs with the resulting equivalent KDFs. I have also been able to use these results to see how different KDF data types appear in the Previewer's book renderer. That, combined with a few instances of unencrypted KFX files obtained from Amazon, has allowed me to have a pretty good idea of how KFX works, at least for reflowable e-books.

I have written some code to convert KDF/KFX back into EPUB in order to test my assumptions. That isn't anywhere near usable as a general function, just a proof of concept. I posted an example result earlier.

I don't see conversion of KFX to EPUB as being a very useful capability. First Amazon encrypts KFX books, even those that are available DRM-free in KF8. The only books that could be converted from KFX to EPUB are those you converted from EPUB to KFX in the first place. Even if decryption eventually becomes possible, KFX is far inferior to KF8 as a source for archival or conversion. The original EPUB internal HTML semantic markup is lost in the conversion of EPUB to KFX.

For example, in KFX conversion <p>, <div>, <h1>, and <blockquote> all produce the same 'text' data type with any differences reflected in the formatting style. The same goes for <b>, <em>, <i>, <small>, <span>, <strike>, <strong>, <sub>, <sup>, etc., which all become just the equivalent css styles applied to text. There is no way to tell from the result whether the original EPUB used these different tags or just used css for formatting. Also values expressed in px or pt units tend to be converted to roughly equivalent em units in KFX.

Hey mate. So I have found the temp directory with the converted book.kdf file but simply renaming this to book.kfx or book.azw8 or whatever and sideloading it to my kindle does not open the book, or list the book at all. Opening the file in calibre doesn't work either. A few pages back you listed a method where you would zip the file, and rename the zip file, but I'm unable to change the zip files file extension from.zip to.kfx or anything! Is this what I'm supposed to do to have a working sideloadable book? Please help, and thanks! Much love

[cerem0ny]

Oops, I didn't read your post properly at the time and have since realised you were describing something different... my bad! My understanding now is that converting kdf to kfx isn't particularly easy to do as well! Shame!

[AaronShep]

Originally Posted by Notjohn

Or with the Mac?

Apple and Amazon have never played well together.

Notjohn, if Amazon delivers a Beta for use on the Mac that doesn't work on the Mac, I don't know how Apple can be blamed for that. It can only be the fault of the programmers. If Apple breaks it later with an update, that's a different story.

But the problem with the Previewer in the past has always been that it wasn't a proper Mac program at all. Instead, it used Java and X11, both of which have been deprecated on the Mac for quite some time. I always felt the Previewer was a shameful kludge on Amazon's part.

[jhowell]

I am continuing to work on a KFX conversion output plugin for calibre based on Kindle Previewer 3.0. I would like the plugin to support Mac OS so I am asking here for some help with that.

Under Windows the EPUB to KDF conversion function is activated by the Previewer running:

jre\bin\java.exe lib/EpubToKFXConverter-1.0.jar;lib/* com.amazon.kfxconverter.app.KFXGenApp ...

With a working directory of:

%UserProfile%\AppData\Local\Amazon\Kindle Previewer 3\lib\fc

I need to know the equivalent working directory and command line for Mac OS. And also whether there is anything else unusual about running the Previewer under that OS, such as environment variables that need to be set.

Thanks for any help from Mac OS gurus.