[Conversion Plugin] KFX Input

[jhowell]

KFX Conversion Input Plugin

The KFX Input plugin allows books in Amazon KFX format to be converted to other e-book formats, such as EPUB, for archival and for reading on alternate devices. The plugin can convert from several KFX variants: Monolithic KFX, KFX-ZIP, or KPF.

Amazon uses KFX format for the delivery of most e-books to customers. It supports advanced features, including Enhanced Typesetting and Page Flip, on Amazon apps and devices. More information about the KFX format can be found in the KFX Format thread and the MobileRead Wiki (KFX).

Books delivered by Amazon in KFX format always contain DRM. Even books that the publisher has specified should not use DRM will be encrypted, although in that case the book will not be locked to a single device.

Amazon's KF8 (azw3) format is superior to KFX as a source for conversion. This plugin has limited usefulness as long as KF8 is still easily obtainable. Some cases where KF8 is not available are Arabic and Indic language books/dictionaries, books produced using Kindle Create, Kindle in Motion books with animation, and Interactive Textbooks with A/V media. In most other cases KFX can be avoided. See the thread Dealing with Kindle for PC/Mac 1.19 and KFX in calibre for details.

Amazon's Kindle Textbook Creator and Kindle Create software export books in KPF (Kindle Package Format), a variant of KFX. This plugin allows conversion of KPF, so that the author has greater control over the finished book.


Installation:

Install the plugin and restart calibre as described in the Introduction to plugins thread.

The latest release of the DeDRM Tools calibre plugin includes KFX support. To allow books in KFX format to be imported that plugin must also be installed and configured because all books delivered by Amazon in KFX format have DRM encryption. See Apprentice Harper's GitHub.


Importing KFX books into calibre for conversion:

Spoiler:

Users of this plugin must manually import KFX books into calibre. They cannot be transferred using the "Device" view.

First, the main KFX file for the book to be imported must be located:

• For Kindle for PC/Mac version 1.19 or greater the "My Kindle Content" folder will contain a sub-folder for each book, named with the ten character ASIN of the book followed by "_EBOK". The ASIN for each book is shown in the "Product details" of the book's Amazon web page. The main file for the book will have an ".azw" extension, regardless of whether or not it is in KFX format. When locating books it may be helpful to sort "My Kindle Content" by date so that the newest book sub-folders appear first.
  
  The location of the "My Kindle Content" folder can be found from within the Kindle for PC program by selecting the menu choices Tools, Options..., Content.
  
• On e-ink Kindle devices connected via USB, the main file for each book is contained in the "documents" folder. These contain both the title of the book and the ASIN. Books in KFX format will have a ".kfx" extension.
  
• Import is possible from Fire tablets and the Kindle for Android app for those books that the publisher has chosen to not lock to a specific device using DRM. Import may be difficult or impossible for books locked with DRM on these platforms. (Using Kindle for PC/Mac is usually a better alternative since it is simpler and more reliable.)
  
  The location of the main book folder can vary somewhat by device, but it will be something like "/sdcard/Android/data/com.amazon.kindle/files" or "/storage/sdcard1/Android/data/com.amazon.kindle/files/Books/amzn1.account.xxxxxx". Each book is contained in a separate sub-folder named with the ten character ASIN of the book, holding files with the extension ".kfx" along with others. Any of these .kfx files can be chosen as the main file of the book.
  
• No workable import method has been discovered for the Kindle for iOS app.

Once located, the main file can be imported to calibre using "Add Books" function ("Add books from a single directory") or using drag-and-drop.

Be patient. The import process is complex and may take a minute or more to complete for a large book.

If successful the result will will appear as KFX format in the calibre library. This can be sideloaded to Kindle apps & devices, viewed using Kindle Previewer 3, or converted to other e-book formats using this plugin.

If the resulting format in calibre is KFX-ZIP instead of KFX then the import was unsuccessful. Information on the reason for import failure can be obtained from a log file. Click the calibre Preferences drop-down menu and choose "Restart in debug mode". Re-import the book and then close calibre. A log file will appear.

For help with DRM removal problems raise an issue at Apprentice Harper's GitHub or make a comment at Apprentice Alf's Blog. Detailed instructions for DRM removal are not allowed on MobileRead.


Import - Behind the scenes:

Calibre handles all e-book formats as a single file, however books in KFX format are usually composed of multiple KFX container files. When the main file of a book in KFX format is imported in calibre a File Type Plugin contained within the KFX Input plugin will automatically locate the rest of the files that make up that book and gather them into a KFX-ZIP archive (a ZIP file with the extension ".kfx-zip".)

As an alternative a KFX-ZIP archive can be created manually. To do this locate the files/folders containing the book's contents, combine them into a ZIP file, change the extension of the ZIP file from .zip to .kfx-zip, and import that into calibre instead of the main book file.

If the DRM removal plugin with KFX support is properly installed and configured it will process the KFX-ZIP and remove its DRM.

The KFX-ZIP archive will then be processed again by the KFX Input plugin to package the content into a single monolithic KFX file.

Viewing books in KFX format:

Spoiler:

To view a KFX book within calibre right click the book entry in the calibre library and from the menu choose "View" and then "View with calibre E-book viewer". This will be slow since the book will be converted from KFX to EPUB each time it is viewed this way. One alternative is convert to EPUB first and then view that.

Another way to view KFX is using Kindle Previewer 3. See "Open KFX with Kindle Previewer 3".

Converting from KFX or KPF to other formats:

Spoiler:

The KFX Input plugin is activated by selecting either (Monolithic) KFX, KFX-ZIP, or KPF as the input format when converting books in calibre. A command line interface is also available. (See below.)

Monolithic KFX format will be produced when a KFX book is successfully imported into calibre. See "Importing KFX books into calibre" above for instructions. Monolithic KFX can also be produced using the KFX Output plugin.

Note: Monolithic KFX is different from the files delivered by Amazon with a ".kfx" extension. Those contain only part of the book's content and cannot be converted individually.

This plugin will fail to convert files that contain unexpected data, such as new or uncommon KFX features. You can force conversion to proceed despite these errors by selecting the option "Allow conversion to complete even if errors are detected" in the KFX Input tab of the conversion dialog. However, in that case the results of conversion may be incomplete or incorrect.

If the option to ignore errors is selected or if the results of a conversion appear to be incorrect, it is recommended that the conversion job log be checked for errors and warnings. Please report problems in this MobileRead thread.

You can view the conversion job log after an error occurs to see additional information on why the conversion failed. To access the log, either click on the "Jobs: 0" label in the lower right corner of the calibre window or use the Alt-Shift-J keyboard shortcut to activate the Jobs dialog. Select the failed conversion job from the list (the most recent will be first) and click the "Show job details" button. To copy the log to the clipboard under Windows, click within the log text then press Ctrl-A (select all) and Ctrl-C (copy). You can then paste it wherever you like.

Reading book metadata:

Spoiler:

The KFX Input plugin also contains a Metadata Reader plugin, which will extract book metadata and cover images on import.

It is also activated by the "Set metadata for the book from the selected format" and "Set the cover for the book from the selected format" buttons in the single Edit Metadata dialog. ("Set metadata from the e-book files" and "Change cover - Set from e-book files" in bulk Edit Metadata.)

The supported fields are: title, author, language, publisher, publication date, description, cover, and "mobi-asin" identifier (ASIN).

Command Line Interface:

Spoiler:

The KFX Input plugin command line interface bypasses the calibre conversion pipeline and can result in an EPUB that more closely matches the formatting of the original book. Conversion using this method is required for books with special formatting, such as fixed layout comics.

usage: calibre-debug -r "KFX Input" -- [-h] [-e EPUBVERSION] infile [outfile]

Convert e-book from KFX to EPUB

positional arguments:
  infile                Pathname of the .azw8, .kfx, .kfx-zip, or .kpf file to
                        be converted to .epub
  outfile               Optional pathname of the resulting .epub file

optional arguments:
  -h, --help            show this help message and exit
  -e EPUBVERSION, --epubversion EPUBVERSION
                        EPUB version to generate: 2.0, 3.0, 3.1

Limitations:

Spoiler:

The KFX format is undocumented and has changed over time. Books containing new or unknown features may fail to convert or result in incorrect content or formatting.

Some specialized variants of KFX cannot be converted using this plugin, including Kindle in Motion, Magazines, and dictionaries.

KFX is more highly processed than Amazon's KF8 (azw3) format, causing some details in the markup of the original e-book provided by the publisher to be lost during conversion. The exact HTML tags and styles originally used in the book are not retained in KFX format. However, all of the text and most of the formatting should be preserved. Images may be reduced in resolution or converted to black and white to match the capabilities of the device to which they were delivered. These limitations make KFX less suitable for archival and conversion when compared with KF8.

This plugin only supports KFX files without DRM. Removal of DRM is not within the scope of this plugin.

This plugin does not contain a copy of the KFX symbol table. Some error messages may refer to KFX symbols in numeric form (such as $123). Interpreting these is not within the scope of this plugin.

This plugin requires calibre version 2.0 or later. Version 2.74 or later is required for automatic import of KFX format. Version 3.9 or later is required for conversion of books from e-ink Kindles containing JPEG-XR images.

Version History:

Spoiler:

Version 1.4.0 - 21 May 2018

Fix error when converting KPF files produced by Kindle Create version 1.6: CorruptError: malformed database schema (?)

Do not include extra metadata in OPF that could lead to duplicate cover pages when converting comics.

Support additional KFX features.

Version 1.3.0 - 09 Apr 2018

Fix error during conversion of books with referenced fonts that are not present: "'unicode' object has no attribute 'list_symbols'"

Support additional KFX features.

Version 1.2.0 - 02 Apr 2018

Added an option to proceed with conversion even if unknown or incorrect content is detected.

Handle additional features of KPF format produced by Kindle Create.

Fix KFX-ZIP archives not being created for e-book samples.

Fix debug message logging when creating KFX-ZIP archives.

Internal restructuring and performance improvements.

Version 1.1.0 - 31 Jan 2018

Added a built-in File Type Plugin to automatically gather the component files of a book and produce KFX-ZIP format during book import. This eliminates the need for the user to produce KFX-ZIP files manually, simplifying the process of importing KFX books into calibre. (DRM removal must still be accomplished by other means, prior to import.)

Added another built-in File Type Plugin to combine the components of a KFX-ZIP into monolithic KFX format. This process will only succeed if all of the components of the book are present in the KFX-ZIP and do not contain DRM. This makes it easier to determine whether on not the import was successful since a successful import will result in KFX format and an unsuccessful one will result in KFX-ZIP format.

Added a Metadata Reader Plugin to extract KFX book metadata. This allows the title, author, and other metadata to appear in the calibre book entry when importing a KFX book.

Ignore approximate page numbers created by the KFX Output plugin.

Handle changes to KPF format produced by Kindle Create 1.3.30.0 and Kindle Previewer 3.17.1.

Fix KPF format not being accepted for conversion using the GUI interface.

Allow conversion of additional KFX features.

Improve detection of unsupported KFX features.

Version 1.0.1 - 16 Oct 2017

Remove extraneous files that were inadvertently included in the initial release.

Version 1.0.0 - 16 Oct 2017

Initial release.

[jhowell]

Reserved post

[odamizu]

Awesome! Thank you, jhowell, for all your work unraveling KFX and making it more accessible to all!

[jhowell]

Originally Posted by odamizu

Awesome! Thank you, jhowell, for all your work unraveling KFX and making it more accessible to all!

You're welcome.

I know that this plugin isn't especially useful right now given the continued availability of KF8 and the lack of automatic DRM removal for KFX. But I thought it would be best to be prepared in case this capability is needed in the future.

[odamizu]

Originally Posted by jhowell

... I thought it would be best to be prepared in case this capability is needed in the future.

Absolutely! Hopefully KF8 and EPUB will continue to be available and accessible for a long time to come, but it gives me great comfort knowing your plug-in (and the Pascal decrypter) exists, just in case. for all your time and effort.

[Terisa de morgan]

Thank you very much, perhaps some of the things which you say they limit its usefulness will disappear in a near future.

[mbovenka]

Originally Posted by odamizu

Absolutely! Hopefully KF8 and EPUB will continue to be available and accessible for a long time to come, but it gives me great comfort knowing your plug-in (and the Pascal decrypter) exists, just in case. for all your time and effort.

+1. Not that I expect EPUB to go anywhere, as Amazon has no control over it, but KF8 might.

[odamizu]

Originally Posted by mbovenka

+1. Not that I expect EPUB to go anywhere, as Amazon has no control over it, but KF8 might.

Amazon isn't the only one we have to worry about. Adobe has their hardened DRM. So far ADE 2.0.1 still works for most Adobe epubs, but that could change.

[pdurrant]

Thank you for this most impressive bit of work.

[Iris-maybe]

I do not think this is unuseful right now since Epubor has update this software to remove Kindle KFX DRM yesterday. Unfortunately, they can not convert KFX. This is really what I need. Thanks for your contribution.

[JSWolf]

epubor is a rip-off of Calibre (overall).

[odamizu]

Originally Posted by JSWolf

epubor is a rip-off of Calibre (overall).

And epubor will likely soon rip-off jhowell's awesome KFX conversion plug-in

[JSWolf]

Originally Posted by odamizu

And epubor will likely soon rip-off jhowell's awesome KFX conversion plug-in

I would not be surprised if they do.

[issybird]

The rest of this conversation has been moved to a new thread here, per request. Please continue the conversation along those lines there; this thread is for the support of the KFX Input plugin only.

[luisdent]

I have decent luck so far, but one book "inferno" by dan brown won't convert to kfx. It seems to be any image files from what I can tell. If I removed the first graphic symbol on one of the pages, the next time i convert it errors on a further page... Any way to keep the graphic elements and still convert?