[Conversion Plugin] KFX Input

[jhowell]

KFX Conversion Input Plugin

The KFX Input plugin allows books in Amazon KFX format to be imported and converted to other e-book formats, such as EPUB, for archival and for reading on alternate devices. The plugin can convert from several KFX variants: Monolithic KFX, KFX-ZIP, or KPF.

Amazon uses KFX format for the delivery of most e-books to customers. It supports advanced features, including Enhanced Typesetting and Page Flip, on Amazon apps and devices. More information about the KFX format can be found in the KFX Format thread and the MobileRead Wiki (KFX).

Amazon's KF8 (azw3) format is superior to KFX as a source for conversion. This plugin has limited usefulness as long as KF8 is still easily obtainable. Some cases where KF8 is not available are Arabic and Indian language books/dictionaries, books produced using Kindle Create, Kindle in Motion books with animation, and Interactive Textbooks with A/V media. In most other cases KFX can be avoided. See the thread Dealing with Kindle for PC/Mac 1.19 and KFX in calibre for details.

Books delivered by Amazon in KFX format always contain DRM. Even books that the publisher has specified should not use DRM will be encrypted, although in that case the book will not be locked to a single device.

Books in KFX format delivered by Amazon are a bundle of files composed of an encrypted main container, a metadata container, auxiliary containers (zero or more) and a DRM voucher. Those each contain only part of the book's content and cannot be converted individually. KFX container files typically have a ".kfx" extension, though other extensions (".azw", ".azw8", and ".azw9") are used by some software.

This plugin will combine the individual files that make up a KFX book into a single file (monolithic KFX format) when a KFX book is successfully imported into calibre. See "Importing KFX books into calibre" above for instructions. Monolithic KFX can also be produced using the KFX Output plugin. Monolithic KFX can be converted to other formats using this plugin.

Amazon's Kindle Textbook Creator and Kindle Create software export books in KPF (Kindle Package Format), a variant of KFX. This plugin allows conversion of KPF, so that the author has greater control over the finished book.


Installation:

Install the KFX Input plugin and restart calibre as described in the Introduction to plugins thread.

The latest release of the DeDRM Tools calibre plugin includes KFX support. To allow books in KFX format to be imported that plugin must also be installed and configured because all books delivered by Amazon in KFX format have DRM encryption. See Apprentice Harper's GitHub. DRM removal will only be possible for purchased (not borrowed or rented) books in KFX format.

DRM removal for KFX format only works with books from Kindle for PC/Mac versions 1.19 through 1.24.3 and e-ink Kindle devices with firmware versions 5.6.5 through 5.10.1.3. (Older versions do not support KFX and newer versions use a new, unsupported DRM scheme.)


Importing KFX books into calibre for conversion:

Spoiler:

Users of this plugin can transfer books in KFX format from an e-ink Kindle running firmware prior to version 5.10.2 using the calibre "Device" view. The plugin tweak "kfx_input_set_format_virtual" must be set to False to enable this type of import. It can be used in cases where the user's Kindle device has compatible firmware.

In addition KFX books can be manually imported into calibre using the following procedure. First, the main KFX file for the book to be imported must be located:

• For Kindle for PC/Mac version 1.19 or greater the "My Kindle Content" folder will contain a sub-folder for each book, named with the ten character ASIN of the book followed by "_EBOK". The ASIN for each book is shown in the "Product details" of the book's Amazon web page. The main file for the book will have an ".azw" extension, regardless of whether or not it is in KFX format. When locating books it may be helpful to sort "My Kindle Content" by date so that the newest book sub-folders appear first.
  
  The location of the "My Kindle Content" folder can be found from within the Kindle for PC program by selecting the menu choices Tools, Options..., Content.
  
  Import cannot currently be performed on KFX books downloaded using Kindle for PC/Mac version 1.25 or greater due to new DRM.
  
• On e-ink Kindle devices connected via USB, the main file for each book is contained in the "documents" folder and sometimes in the "Downloads" subfolder of that folder. These files are named with both the title of the book and the ASIN. Books in KFX format will have a ".kfx" extension.
  
• Import is possible from Fire tablets and the Kindle for Android app for those books that the publisher has chosen to not lock to a specific device using DRM. Import may be difficult or impossible for books locked with DRM on these platforms. (Using Kindle for PC/Mac is usually a better alternative since it is simpler and more reliable.)
  
  The location of the main book folder can vary somewhat by device, but it will be something like "/sdcard/Android/data/com.amazon.kindle/files" or "/storage/sdcard1/Android/data/com.amazon.kindle/files/Books/amzn1.account.xxxxxx". Each book is contained in a separate sub-folder named with the ten character ASIN of the book, holding files with the extension ".kfx" along with others. Any of these .kfx files can be chosen as the main file of the book.
  
  Import cannot currently be performed on KFX books downloaded using Kindle for Android version 8.12 or greater and Kindle for Fire 14.12 or greater due to new DRM.
  
• No workable import method is available for the Kindle for iOS app.

Once located, the main file can be imported to calibre using "Add Books" function ("Add books from a single directory") or using drag-and-drop. (The main book file must be in its original folder structure. For example, books from Kindle for PC should be in a subfolder of the "My Kindle Content" folder.)

Be patient. The import process is complex and may take a minute or more to complete for a large book.

If successful the result will appear as KFX format in the calibre library. This can be sideloaded to Kindle apps & devices, viewed using Kindle Previewer 3, or converted to other e-book formats using this plugin.

If the resulting format in calibre is KFX-ZIP or AZW instead of KFX then the import was unsuccessful. If this occurs make sure that you have both the KFX Input plugin and the DeDRM plugin installed, that you restarted calibre after installation, and that the main book file was imported from its original location.

The reason for import failure can be obtained from a log file. Click the calibre Preferences drop-down menu and choose "Restart in debug mode". Re-import the book and then close calibre. A log file will appear.

DRM removal will not function for borrowed or rented books in KFX format. For help with other DRM removal problems raise an issue at Apprentice Harper's GitHub or make a comment at Apprentice Alf's Blog. Detailed instructions for DRM removal are not allowed on MobileRead.


Automatic adding of books:

The "Automatic adding" function of calibre, which adds any book files placed in a specific directory, cannot be used with books in KFX format because each book in composed of multiple files. You must manually add the main file of the book from its original folder location so that the KFX Input plugin can gather all of the files that make up the book.

Alternatively, you can create a ZIP file containing all of the files that make up the book, change its extension from ".zip" to ".kfx-zip", and place that into the folder configured for automatic adding to calibre.


Import - Behind the scenes:

Calibre handles all e-book formats as a single file, however books in KFX format are usually composed of multiple KFX container files. When the main file of a book in KFX format is imported in calibre a File Type Plugin contained within the KFX Input plugin will automatically locate the rest of the files that make up that book and gather them into a KFX-ZIP archive (a ZIP file with the extension ".kfx-zip".)

As an alternative a KFX-ZIP archive can be created manually. To do this locate the files/folders containing the book's contents, combine them into a ZIP file, change the extension of the ZIP file from .zip to .kfx-zip, and import that into calibre instead of the main book file.

If the DRM removal plugin with KFX support is properly installed and configured it will process the KFX-ZIP and remove its DRM.

The KFX-ZIP archive will then be processed again by the KFX Input plugin to package the content into a monolithic KFX file.

Viewing books in KFX format:

Spoiler:

To view a KFX book within calibre right click the book entry in the calibre library and from the menu choose "View" and then "View with calibre E-book viewer". This will be slow since the book will be converted from KFX to EPUB each time it is viewed this way. One alternative is convert to EPUB first and then view that.

Another way to view KFX is using Kindle Previewer 3. See "Open KFX with Kindle Previewer 3".

Converting from KFX or KPF to other formats:

Spoiler:

The KFX Input plugin is activated by selecting either (Monolithic) KFX, KFX-ZIP, or KPF as the input format when converting books in calibre.

By default calibre will reduce the resolution of images during conversion. In the "Page setup" tab the "Output profile" may be set to "Tablet" in order to avoid this.

A command line interface is also available for this plugin. (See below.)

Dealing with conversion errors:

Spoiler:

When importing a book from Kindle for PC/Mac the original file name will have an ".azw" extension. If the import process fails it can result in the book showing AZW format within your calibre library. calibre will treat that as being in a different Kindle e-book format than KFX, leading the error "Failed to convert: (name) This is an Amazon KFX book. It cannot be processed...." when attempting to convert it to another format.

This plugin will fail to convert files that contain unexpected data, such as new or uncommon KFX features. You can force conversion to proceed despite these errors by selecting the option "Allow conversion to complete even if errors are detected" in the KFX Input tab of the conversion dialog. However, in that case the results of conversion may be incomplete or incorrect.

If the option to ignore errors is selected or if the results of a conversion appear to be incorrect, it is recommended that the conversion job log be checked for errors and warnings. Please report problems in this MobileRead thread.

You can view the conversion job log after an error occurs to see additional information on why the conversion failed. To access the log, either click on the "Jobs: 0" label in the lower right corner of the calibre window or use the Alt-Shift-J keyboard shortcut to activate the Jobs dialog. Select the failed conversion job from the list (the most recent will be first) and click the "Show job details" button. To copy the log to the clipboard under Windows, click within the log text then press Ctrl-A (select all) and Ctrl-C (copy). You can then paste it wherever you like.

Reading book metadata:

Spoiler:

The KFX Input plugin also contains a Metadata Reader plugin, which will extract book metadata and cover images on import.

It is also activated by the "Set metadata for the book from the selected format" and "Set the cover for the book from the selected format" buttons in the single Edit Metadata dialog. ("Set metadata from the e-book files" and "Change cover - Set from e-book files" in bulk Edit Metadata.)

The supported fields are: title, author, language, publisher, publication date, description, cover, and "mobi-asin" identifier (ASIN).

Command Line Interface:

Spoiler:

The KFX Input plugin command line interface bypasses the calibre conversion pipeline and can result in an EPUB that more closely matches the formatting of the original book. Conversion using this method is required for books with special formatting, such as fixed layout comics and print replica books.

Instead of converting to EPUB, the resources contained within a book can be extracted as a ZIP archive. This can be used to obtain PDF resources from print replica books. The archive will also contain a file named "book.ion" with the content of the book in ion text format.

The "book.ion" file can be produced containing readable names, instead of numeric symbols, by use of an optional symbol table. The plugin will make use of a catalog file named "kfx_symbol_catalog.ion" containing the definition of the YJ_symbols shared symbol table if it is present in the calibre plugin configuration folder. (This file is not included with the plugin.)

usage: calibre-debug -r "KFX Input" -- [-h] [-e EPUBVERSION] [-u]
                                       infile [outfile]

Convert KFX e-book to EPUB or extract its resources

positional arguments:
  infile                Pathname of the .azw8, .kfx, .kfx-zip, or .kpf file to
                        be processed
  outfile               Optional pathname of the resulting .epub or .zip file

optional arguments:
  -h, --help            show this help message and exit
  -e EPUBVERSION, --epubversion EPUBVERSION
                        EPUB version to generate: 2.0, 3.0, 3.1
  -u, --unpack          Create a ZIP with resources instead of converting to
                        EPUB

Limitations:

Spoiler:

The KFX format is undocumented and has changed over time. Books containing new or unknown features may fail to convert or result in incorrect content or formatting.

KFX is more highly processed than Amazon's KF8 (azw3) format, causing some details in the markup of the original e-book provided by the publisher to be lost during conversion. The exact HTML tags and styles originally used in the book are not retained in KFX format. However, all of the text and most of the formatting should be preserved. Images may have been reduced in resolution or converted to black and white to match the capabilities of the device to which they were delivered.

PDF resources in KFX print replica books may have been split into multiple files. Links from one file to pages that are in another file will be broken.

These limitations make KFX less suitable for archival and conversion when compared with KF8.

This plugin only supports KFX files without DRM. Removal of DRM is not within the scope of this plugin.

Some specialized variants of KFX cannot be converted using this plugin, including Kindle in Motion and dictionaries.

This plugin does not contain a copy of the KFX symbol table. Unpacked book content and error messages may refer to KFX symbols in numeric form (such as $123). Interpreting these is not within the scope of this plugin, however the plugin can make use of a symbol table if provided by the user.

This plugin requires calibre version 2.0 or later. Version 2.74 or later is required for automatic import of KFX format. Version 2.80 or later is required to import books in KFX format from the device view. Version 3.24 or later is required for correct metadata when importing KFX books using the "calibredb add" command.

Version History:

Spoiler:

Version 1.13.0 - 02 Feb 2019

Handle unexpected data found in some print replica textbooks. (Fixes "TypeError: %d format: a number is required, not IonStruct")

Ignore unexpected control characters found in some print replica textbooks.

Support KPF files produced by Kindle Create version 1.20. (Fixes "$538 has extra data: ('yj.semantics.page_number_begin': ...")

Support additional KFX features. (reflow-language-2, reflow-language-expansion-1, reflow-style-12, Ion symbols $136 $137)

Version 1.12.1 - 30 Dec 2018

Support additional KFX feature. (Fixes "Unexpected Ion symbols used: $282")

Version 1.12.0 - 20 Dec 2018

Change the default value of the "kfx_input_set_format_virtual" plugin tweak to False so that by default calibre will not import books in KFX format from e-ink Kindle devices. (Books in KFX format can no longer be imported from the latest Kindle firmware, version 5.10.2.)

Fix incorrect font size used for drop caps in some cases.

Allow combinations of text-decoration line values.

Support additional KFX features. (Fixes "ERROR: Unexpected Ion symbols used: $135, $353", "ERROR: unexpected style events in $270", and "ERROR: Unknown com.amazon.yjconversion feature: yj_mathml-1".)

Version 1.11.0 - 08 Dec 2018

Add plugin tweak "kfx_input_set_format_virtual" that when set to True will prevent import of books in KFX format from e-ink Kindle devices in calibre "Device" view.

Support additional KFX features. (Fixes "ERROR: Unknown com.amazon.yjconversion feature: yj_table-8", "ERROR: Unexpected Ion symbols used: $292, $293, $749, $750, $751, $752", and "ERROR: Unexpected Ion symbols used: $760, $761".)

Ignore extraneous data produced by Kindle Previewer version 3.28.1.

Version 1.10.0 - 11 Nov 2018

Allow books in KFX format to be imported from a Kindle using the device view. Calibre (version 2.80 or higher) will no longer treat KFX as a "virtual format" when this plugin is installed.

Improve handling of properties that combine several others in an equivalent shorthand. (Fixes "ERROR: Setting conflicting property: border-top-style...".)

Support additional KFX features. (Fixes "ERROR: Unknown com.amazon.yjconversion feature: reflow-style-11".)

Version 1.9.0 - 11 Oct 2018

Fix extra text-indent properties from being produced for some books.

Fix incorrect placement of "colspan" in table cells containing only an image. (Fixes "ERROR: Unexpected class_attribute in img: colspan")

Fix uninformative error details provided if a conversion error occurs while attempting to use the calibre viewer on a book in KFX format.

Improve handling of long path names when importing books under Windows.

Support additional KFX features. (Fixes "ERROR: Unknown com.amazon.yjconversion feature: reflow-style-8".)

Version 1.8.0 - 30 Aug 2018

Prevent occasional failures when converting some books from e-ink Kindles that contain problematic JPEG-XR images.

Support additional KFX features produced by Kindle Create version 1.12.

Version 1.7.0 - 13 Aug 2018

Fix incorrect image type sometimes being detected for books that were published containing SVG images.

Prevent conversion failure caused by unexpected content in some KFX books. (Fixes "Unexpected box-align value: justify")

Prevent conversion failure caused by alignment of lists in KFX books. (Fixes "Unexpected box-align found in ol element")

Prevent conversion failure caused by drop caps with zero length, sometimes produced by Kindle Create.

Fix incompatibility with calibre versions before 3.0 introduced in plugin version 1.2.0. (Fixes "AttributeError: 'apsw.Cursor' object has no attribute 'fetchone'" when converting from KPF format.)

Support additional KFX features.

Version 1.6.1 - 20 Jul 2018

Prevent conversion failure when a book contains an unexpected symbol count.

Version 1.6.0 - 20 Jul 2018

When converting to EPUB report in the log if the book contains any PDF resources. This can occur for print replica books.

Allow book content and resources to be unpacked and extracted as a ZIP archive using the command line interface.

Fix bug introduced in version 1.4.0 that causes incorrect styles to be produced for drop caps.

Support additional KFX features.

Version 1.5.0 - 25 Jun 2018

Create KFX-ZIP format during import even if additional files that make up the book cannot be located. This prevents potential confusion when books from Kindle for PC/Mac in KFX format fail to import, previously resulting in AZW format.

Improve error messages associated with unsuccessful import of books in KFX format.

Support books containing unicode characters with code points outside of the basic multilingual plane, such as emoji.

Retain original image file names when converting KPF files produced by Kindle Create.

Prevent conversion failure when unexpected quoting of font names is encountered. Fixes conversion error: "Incorrectly quoted font name: 'arial"

Prevent conversion failure when the OS platform on which the plugin is running cannot be properly determined.

Version 1.4.0 - 21 May 2018

Fix error when converting KPF files produced by Kindle Create version 1.6: CorruptError: malformed database schema (?)

Do not include extra metadata in OPF that could lead to duplicate cover pages when converting comics.

Support additional KFX features.

Version 1.3.0 - 09 Apr 2018

Fix error during conversion of books with referenced fonts that are not present: "'unicode' object has no attribute 'list_symbols'"

Support additional KFX features.

Version 1.2.0 - 02 Apr 2018

Added an option to proceed with conversion even if unknown or incorrect content is detected.

Handle additional features of KPF format produced by Kindle Create.

Fix KFX-ZIP archives not being created for e-book samples.

Fix debug message logging when creating KFX-ZIP archives.

Internal restructuring and performance improvements.

Version 1.1.0 - 31 Jan 2018

Added a built-in File Type Plugin to automatically gather the component files of a book and produce KFX-ZIP format during book import. This eliminates the need for the user to produce KFX-ZIP files manually, simplifying the process of importing KFX books into calibre. (DRM removal must still be accomplished by other means, prior to import.)

Added another built-in File Type Plugin to combine the components of a KFX-ZIP into monolithic KFX format. This process will only succeed if all of the components of the book are present in the KFX-ZIP and do not contain DRM. This makes it easier to determine whether on not the import was successful since a successful import will result in KFX format and an unsuccessful one will result in KFX-ZIP format.

Added a Metadata Reader Plugin to extract KFX book metadata. This allows the title, author, and other metadata to appear in the calibre book entry when importing a KFX book.

Ignore approximate page numbers created by the KFX Output plugin.

Handle changes to KPF format produced by Kindle Create 1.3.30.0 and Kindle Previewer 3.17.1.

Fix KPF format not being accepted for conversion using the GUI interface.

Allow conversion of additional KFX features.

Improve detection of unsupported KFX features.

Version 1.0.1 - 16 Oct 2017

Remove extraneous files that were inadvertently included in the initial release.

Version 1.0.0 - 16 Oct 2017

Initial release.

[jhowell]

Reserved post

[odamizu]

Awesome! Thank you, jhowell, for all your work unraveling KFX and making it more accessible to all!

[jhowell]

Originally Posted by odamizu

Awesome! Thank you, jhowell, for all your work unraveling KFX and making it more accessible to all!

You're welcome.

I know that this plugin isn't especially useful right now given the continued availability of KF8 and the lack of automatic DRM removal for KFX. But I thought it would be best to be prepared in case this capability is needed in the future.

[odamizu]

Originally Posted by jhowell

... I thought it would be best to be prepared in case this capability is needed in the future.

Absolutely! Hopefully KF8 and EPUB will continue to be available and accessible for a long time to come, but it gives me great comfort knowing your plug-in (and the Pascal decrypter) exists, just in case. for all your time and effort.

[Terisa de morgan]

Thank you very much, perhaps some of the things which you say they limit its usefulness will disappear in a near future.

[mbovenka]

Originally Posted by odamizu

Absolutely! Hopefully KF8 and EPUB will continue to be available and accessible for a long time to come, but it gives me great comfort knowing your plug-in (and the Pascal decrypter) exists, just in case. for all your time and effort.

+1. Not that I expect EPUB to go anywhere, as Amazon has no control over it, but KF8 might.

[odamizu]

Originally Posted by mbovenka

+1. Not that I expect EPUB to go anywhere, as Amazon has no control over it, but KF8 might.

Amazon isn't the only one we have to worry about. Adobe has their hardened DRM. So far ADE 2.0.1 still works for most Adobe epubs, but that could change.

[pdurrant]

Thank you for this most impressive bit of work.

[Iris-maybe]

I do not think this is unuseful right now since Epubor has update this software to remove Kindle KFX DRM yesterday. Unfortunately, they can not convert KFX. This is really what I need. Thanks for your contribution.

[JSWolf]

epubor is a rip-off of Calibre (overall).

[odamizu]

Originally Posted by JSWolf

epubor is a rip-off of Calibre (overall).

And epubor will likely soon rip-off jhowell's awesome KFX conversion plug-in

[JSWolf]

Originally Posted by odamizu

And epubor will likely soon rip-off jhowell's awesome KFX conversion plug-in

I would not be surprised if they do.

[issybird]

The rest of this conversation has been moved to a new thread here, per request. Please continue the conversation along those lines there; this thread is for the support of the KFX Input plugin only.

[luisdent]

I have decent luck so far, but one book "inferno" by dan brown won't convert to kfx. It seems to be any image files from what I can tell. If I removed the first graphic symbol on one of the pages, the next time i convert it errors on a further page... Any way to keep the graphic elements and still convert?