[Conversion Output] KFX Output

[jhowell]

KFX Conversion Output Plugin

The purpose of the KFX Output plugin is to convert e-books into KFX files that can be sideloaded to newer Kindle apps and devices. This allows books obtained from sources other than Amazon to be read with Enhanced Typesetting, multi-thumbnail Page Flip, non-justified alignment, continuous scrolling, page curl, improved manga experience, and Guided View for comics.

(Note: If you only want somewhat better typography when reading Kindle books then the Hyphenate This! plugin should be considered as an alternative to using KFX. It is less difficult to use and the book format it works with is compatible with more devices and software.)


Background:

Amazon has added support for several advanced features in newer kindle reading apps and firmware. Enhanced Typesetting allows improved rendering of e-book content using hyphenation, kerning and ligatures. Page Flip, improved manga experience (fast continuous page turns, pan/zoom, crop margins, page spreads), and Guided View for comics provide improved navigation within a book's contents. Only e-books in the proprietary KFX format support these features and currently only Amazon sourced e-books downloaded directly from Amazon to devices are available in this format. Amazon does not provide a way for Kindle users to convert their own books to KFX format.

Amazon Kindle Previewer 3 is software that takes books in EPUB format and renders them with Enhanced Typesetting. It does this by converting the EPUB file to KPF, which has the same content as KFX but in a different container. This plugin is able to invoke the Previewer and reformat its output to produce KFX files suitable for sideloading.

The plugin performs the following steps during conversion:

• Convert from the original e-book format to EPUB.
• Correct some common EPUB content issues that cause problems for the Previewer.
• Use the Amazon Kindle Previewer to convert from EPUB to KPF.
• Repackage the KPF data into a KFX container.

KFX books delivered by Amazon are a bundle of files composed of an encrypted main container, a metadata container, auxiliary containers (zero or more) and a DRM voucher. The plugin instead generates an equivalent unencrypted monolithic KFX file, which is supported by Amazon reading apps and devices when sideloaded.

A more in-depth discussion of the KFX format can be found in the KFX Format thread and the MobileRead Wiki (KFX).

KPF (Kindle Package Format) is a ZIP archive. The main components are a descriptive KCB file in JSON format and a KDF file containing the main book content in an SQLITE database. KPF can be produced by Kindle Previewer 3 (in unzipped form), Kindle Textbook Creator, and Kindle Create.


Installation:

The Amazon Kindle Previewer 3 should be installed in the default location under the same user account as is being used to run calibre. It is available from http://www.amazon.com/gp/feature.html/?docId=1003018611 for Windows and Mac. Version 3.29.1 (Jan 10, 2019) is recommended for best results. (Versions below 3.27 are not compatible with this plugin. Versions newer than 3.29.1 have not been tested and might not be compatible.)

Install the KFX Output plugin and restart calibre as described in the Introduction to plugins thread.

You can tell if the plugin/Previewer combination is functioning properly by attempting to convert a book that is known to work. One example is this edition of Frankenstein by Mary Shelley.


Configuration:

Configuration of the plugin is done in the KFX Output tab in the conversion options dialog. Defaults for future conversions can be set under Preferences, Output Options, KFX Output. These options are described in the Usage section.

In addition "tweaks" can be configured to set some lesser used options that apply to all conversions. Set them by going to Preferences->Advanced->Tweaks->Plugin tweaks. Tweaks are entered as Python variable assignments.

The "kfx_output_previewer_path" plugin tweak can be used to set the location where Kindle Previewer has been installed in cases where the default location was not used. (Windows paths require double backslashes.) For example:

kfx_output_previewer_path = 'C:\\Program Files (x86)\\Kindle Previewer 3'

A unique ASIN can be forced to be created during conversion even if an ASIN identifier or internal book metadata is present by setting the "kfx_output_ignore_asin_metadata" plugin tweak.

kfx_output_ignore_asin_metadata = True

Usage:

The plugin is activated by selecting KFX as the output format when converting books in calibre. An alternative command line interface is available for advanced users. (See below.)

By default the converted book will be marked as an Amazon purchased book. This enables the most features in Amazon reading apps and devices, but because the book was not actually purchased from Amazon unexpected behavior may sometimes result. The book can be marked as a personal document instead using the "Create personal document instead of book" conversion option. This may be of use if odd behavior is experienced.

If the calibre book being converted has a 10-character mobi-asin, amazon, or asin identifier that starts with B then this will be used as the ASIN of the KFX file created. Otherwise a unique ASIN will be generated. A unique ASIN can be forced to be created even if an ASIN identifier or internal book metadata is present by setting the "kfx_output_ignore_asin_metadata" plugin tweak. A unique ASIN is also always generated for personal documents regardless of the identifiers present.

Kindle Previewer 3 produces a lot of output during the conversion process, most of it useless. The plugin extracts error messages and shows them in the conversion job log. The full contents of the Previewer conversion logs can be saved by selecting "Show full Kindle Previewer conversion logs" in the conversion options. This might occasionally be useful in diagnosing a failure to convert a book.

Page numbers contained in an NCX pageList, NAV page-list, or page-map.xml within the source EPUB will be converted. (The CLI must be used to preserve pages from a NAV page-list since calibre discards that during conversion.) Amazon APNX is not supported as a page number source.

An option is available to create approximate page numbers in cases where page numbers are not present in the source format. The lookup name of a custom column holding the desired number of pages for each book may be specified, otherwise the page count will be determined automatically from the amount of book content.

It is recommended that the "Tablet" output profile be selected in the Page setup section of conversion options to avoid having images downscaled by the calibre conversion pipeline.

The resulting KFX file can be viewed using Kindle Previewer 3. However, the Previewer only supports the file extension ".azw8" for opening KFX files instead of ".kfx". As a workaround, "Open KFX with Kindle Previewer 3.bat" can be installed in Windows to allow files created by this plugin with a .kfx extension to be viewed more easily. (See this post for details.)

KFX files created by the plugin must be sideloaded (transferred into the target device's storage). Calibre version 2.80 or later supports sideloading and viewing metadata for books in KFX format on connected Kindle devices. Calibre version 3.29 or later will also create cover thumbnails when KFX books are sent to a Kindle device.

to sideload using calibre, you need to configure KFX as a format that can be sent to that device. With your device connected click on the "Device" drop-down menu and choose "Configure this device". Under "Select available formats and their order for this device", add a checkmark to "kfx" and use the arrow buttons move it up/down and set its priority relative to the other formats.

Sideloading may also be done manually. The procedure for doing this varies by device type:

• Kindle e-ink devices: Instructions
• Amazon Fire tablets: Instructions (KFX files must be marked as ebook instead of personal document if placed in the Books folder of Internal Storage. The "Docs" app must be used instead of "Books" to view personal documents.)
• Kindle for iOS app: See "About File Sharing on iPhone, iPad, and iPod touch" (You will not be able to use iCloud or the Files app for this transfer.)
• Kindle for Android app: Copy the file to the kindle or Books folder within main internal storage of the device (/sdcard).

Showing Cover Thumbnail Images on E-Ink Kindle Devices:

Spoiler:

Kindle apps will show sideloaded KFX book covers in the thumbnail view, but e-ink Kindle devices may sometimes show a generic thumbnail instead.

Starting with Kindle firmware version 5.8.5 cover thumbnails are no longer supported for personal documents. If thumbnails are desired, use the default conversion option to create a "book" instead.

Starting with calibre version 3.29, when the "Send to device" function is used to send a KFX book to an e-ink Kindle, a cover thumbnail image will also be sent. Use of older calibre versions or sideloading books to the Kindle without using calibre may result in missing cover thumbnails. (The ExtractCoverThumbs program can be used as an alternative to calibre for producing thumbnails. See the MobileRead thread.)

Amazon identifies books that it sells using a ten character Amazon Standard Identification Number (ASIN). If an ASIN is set in a KFX book then the e-ink Kindle device will automatically download the corresponding cover thumbnail image from Amazon's servers when the book is sideloaded. This will replace any thumbnail that may have been add by calibre when the book was sent to the device.

Setting an ASIN can be used to force the Kindle to download cover thumbnails from Amazon in cases when the calibre "Send to device" function is not being used to transfer books. Your Kindle will need to have a Wi-Fi or 3G connection for this to work properly.

In order to set an ASIN:

• Search amazon.com with a web browser for an Amazon Kindle book equivalent to the book you are converting and bring up its page. The Amazon web page must be for a Kindle-edition, a print edition will not work. If there is no equivalent book this procedure will not work. (If your Kindle is registered with an account at a country-specific Amazon site, such as amazon.co.uk, then you must use ASINs from the same site. Each Amazon web site uses its own ASINs.)
  
• The web browser address bar will have something like "https://www.amazon.com/gp/product/B005MZN2B2/...". The ASIN for the book is the 10-character string that begins with a "B".
  
• In calibre select your book, choose "Edit Metadata", and in the "Ids:" field add "amazon:B005MZN2B2" (without the quotes and using the ASIN you discovered). If there is something already in the "Ids:" field then separate the new information with a comma.
  
  Calibre supports ASINs for country-specific Amazon stores by appending the country code to the identifier name when creating the "amazon" identifier. For example:
  amazon:B0064CPN7I --> https://www.amazon.com/dp/B0064CPN7I
  amazon_uk:B00BAJ6GL2 --> https://www.amazon.co.uk/dp/B00BAJ6GL2
  amazon_de:B016PFA0YO --> https://www.amazon.de/dp/B016PFA0YO
• You can verify that the ASIN is set correctly by selecting the book in calibre and looking at the book details panel (usually on the far right). There should be an "Amazon.com" entry in the "Ids:" section. Click on "Amazon.com" there and your browser will open to the associated web page. Verify that the browser shows the expected book and that it is a Kindle edition, not a print edition. Use "Edit Metadata" to fix the ASIN if not.
  
• Convert the book to KFX and sideload it to your kindle device. After you eject your Kindle from your computer the cover thumbnail will be automatically downloaded from Amazon's servers.

Command Line Interface (CLI):

Spoiler:

The KFX Output plugin has a command line interface for advanced users.

usage: calibre-debug -r "KFX Output" -- [-h] [-a ASIN] [-c] [-d] [-p PAGES]
                                        infile [outfile]

Convert e-book to KFX format

positional arguments:
  infile                Pathname of the .last, .epub, .opf, .mobi, .doc,
                        .docx, .kpf, or .kfx-zip file to be converted
  outfile               Optional pathname of the resulting .kfx file

optional arguments:
  -h, --help            show this help message and exit
  -a ASIN, --asin ASIN  Optional ASIN to assign to the book
  -c, --clean           Save the input file cleaned for conversion to KFX
  -d, --doc             Create personal document (PDOC) instead of book (EBOK)
  -p PAGES, --pages PAGES
                        Create n approximate page numbers if missing from
                        input file (0 for auto)

Conversion of the Most Recent Book from the Kindle Previewer GUI

If the infile name ends with a ".last" extension it signals the plugin to locate the most recent file converted by the Kindle Previewer GUI and use that as the source for conversion to KFX. (A file should not actually exist with this name. The ".last" extension is simply used to activate this function.)

This is useful in cases where the plugin fails to function properly within the calibre GUI.

Examples:

calibre-debug -r "KFX Output" -- anyname.last output.kfx  (Converts most recent book to output.kfx)
calibre-debug -r "KFX Output" -- "my book.last"           (Produces my book.kfx)

Follow these steps to use this function:

• Open a book using the Kindle Previewer GUI. The Previewer should show a dialog box with "Please wait while Kindle Previewer converts your book into Kindle format" followed by another dialog box with "Your book is successfully converted into Kindle format".
  
• If the book opens without showing these dialogs then the Previewer is using a cached copy of the book instead of converting it. In this case, delete the contents of the "My Kindle Books" folder and open the book again in order to force conversion to occur.
  
• Verify that the Previewer shows shows "Enhanced Typesetting: Supported" on the book information screen. This is displayed using Ctrl-I or by selecting View->Book Information via the program menu. If it shows "Not Supported", you will be unable to convert that book to KFX.
  
• Invoke the plugin from a command prompt. For example:
  calibre-debug -r "KFX Output" -- Frankenstein.last

Conversion from EPUB or Word Document

If an EPUB file is used as the infile it will be passed to the Amazon Kindle Previewer for conversion to KPF and then the result will be repackaged as KFX. This bypasses any manipulation of the EPUB by the calibre conversion pipeline. English language MS Word DOC and DOCX files and MOBI files (created using kindlegen version 2.9 without the -dont_append_source option) can also be converted the same way.

Examples:

calibre-debug -r "KFX Output" -- input.epub output.kfx  (Converts input.epub to output.kfx)
calibre-debug -r "KFX Output" -- "my book.epub"         (Produces my book.kfx)

An unzipped EPUB can also be converted by passing the OPF file as input. In that case the OPF is passed directly to the Kindle Previewer for conversion without any fix ups being done.


Conversion from KPF

If a KPF file is used as the infile it is repackaged as KFX. This can be used to convert books and comics produced using Amazon Kindle Create.

Converting Comics and Manga:

Spoiler:

The plugin is able to convert fixed layout comics and manga, however there are a number of limitations and restrictions.

Comics must be formatted as per the Amazon Kindle Publishing Guidelines for conversion to KFX. The source file may be unzipped (OPF) or zipped (EPUB). CBR and CBZ files cannot be used.

Files produced by Amazon's Kindle Comic Creator can usually be converted to KFX. Source files for conversion can be extracted from comics in MOBI or KF8 format using Kindle Unpack, though they may require some editing for successful conversion.

Kindle Comic Converter has a KFX output profile compatible with this plugin.

Comics with Guided View can also be produced using Kindle Create version 1.6 or higher in Early Access mode under Windows. After creating the comic use the Publish option to produce a KPF file and then convert that to KFX using this plugin.

The plugin's command line interface must be used to convert comics. (Using the calibre GUI will produce destructive changes to the source file that will prevent conversion.)

For example, if the root source file of the comic is named mycomicbook.opf it can be converted to KFX using the following command:

calibre-debug -r "KFX Output" -- mycomicbook.opf

Additional comic/manga limitations:

KFX comics sideloaded to an e-ink Kindle display in manga mode, even if they are not manga. Panel view does not function. Various problems may occur if the Kindle is not running the latest firmware. Conversely, manga mode does not function in the Android or iOS Kindle app, but comics with panels will display using Guided View.

The plugin produces each comic as a single KFX file that may sometimes be too large for Kindles to handle. This can cause lock up and return to home screen during reading.

Images quality is sometimes reduced by the conversion process.

See "Dealing with Conversion Errors" for descriptions of some errors that can occur during comic conversion.

Dealing with Conversion Errors

Spoiler:

The plugin corrects some issues in the source EPUB that prevent conversion by Previewer: non-UTF8 character encoding, obfuscated fonts, language "UND", missing/duplicated title, and a few others. But many potential problems cannot be fixed automatically.

The Previewer often fails when something unexpected is encountered in the contents of the source EPUB. Some examples of things that may cause a conversion failure are:

• Incorrect HTML or CSS files. (The calibre editor can often be used to detect and correct these problems. EPUBs can be validated using EpubCheck.)
• Large tables.
• Incorrect or unusual style properties or values.
• SVG graphics and images with internal errors.
• Comics and manga that do not follow the Amazon Kindle Publishing Guidelines.
• Miscellaneous things that trigger bugs in the Previewer software.

If a conversion error occurs the plugin attempts to capture the most relevant error message from temporary log files produced by the Previewer. When this occurs an error dialog box will pop up in calibre. The error messages produced are sometimes cryptic, but better than nothing.

The Kindle Previewer provides guidance to help with fixing problems that can prevent conversion to KFX or produce incorrect conversion results. This information will be shown in conversion job log. In some cases additional guidance will be added by the plugin.

You can view the conversion job log after an error occurs to see additional information on why the conversion failed. To access the log, either click on the "Jobs: 0" label in the lower right corner of the calibre window or use the Alt-Shift-J keyboard shortcut to activate the Jobs dialog. Select the failed conversion job from the list (the most recent will be first) and click the "Show job details" button. To copy the log to the clipboard under Windows, click within the log text then press Ctrl-A (select all) and Ctrl-C (copy). You can then paste it wherever you like.

When Previewer errors occur it can be a tedious trial-and-error process to get a book to convert to KFX. One procedure is to use the calibre editor to selectively remove pieces of the source book until conversion is successful. This will narrow down what part of the book is causing the Previewer failure.

If a conversion fails and you feel that it may be caused by a problem with the plugin itself, rather than the Kindle Previewer, please report it in this thread along with the associated error message and log. If posting log contents please enclose them in [spoiler]...[/spoiler] tags.

Updating book metadata

Spoiler:

The metadata and cover images in KFX files produced by this plugin can be updated by a Metadata Writer Plugin that is built into this plugin. This is activated in calibre using the "Embed metadata" function, "Save to disk" function (with the "Update metadata in saved copies" option enabled), "Send to device", or by the "calibredb embed_metadata" command line interface. Metadata plugboards can be used. (The "Polish books" function will not update KFX metadata since it supports only the AZW3 and EPUB formats.)

The supported fields are: title, author, language, publisher, publication date, description, cover, and ASIN.

Only the first author name is used by the Previewer when converting to KFX, however updating KFX metadata supports multiple author names. Each author name will be individually converted to sorted form. (The author_sort field is not used.)

The ASIN will be set from a "mobi-asin", "amazon", or "asin" identifier that has 10 alphanumeric characters starting with "B". If an ASIN is set on a personal document (PDOC) the content type will be changed to book (EBOK).

Approximate page numbers can also be produced or updated based on the default settings for conversion to KFX format.

Updating of KFX metadata can be disabled by disabling the "Set KFX metadata" plugin in the "Metadata writer plugins" category of plugin preferences.

Limitations:

Spoiler:

The KFX format is undocumented. This plugin might sometimes produce incorrectly structured KFX files that could cause a slowdown or crash of your Kindle device or app when sideloaded. Use at your own risk!

Many books cannot be successfully converted to KFX using the Kindle Previewer. The formatting requirements for KFX are undocumented.

Dictionaries in KFX format cannot be created using this plugin.

KFX files cannot be sent using Send to Kindle by E-mail or using Send to Kindle for PC/Mac. They must be sideloaded. Sideloaded books are not backed up to your Amazon account and will not sync reading position, notes and highlights with your account or between your Kindle apps and devices.

KFX files produced by this plugin are uncompressed and so will usually be larger than other e-book formats. (KFX files for books purchased from Amazon include data compression in conjunction with their DRM.) Files produced by this plugin may sometimes be too large for Kindles to handle. This can cause lock up and return to home screen during reading.

The X-Ray feature is not currently available with sideloaded KFX books. (Books in KFX format can make use of X-Ray if they are marked as books instead of personal documents, but there is currently no X-Ray file creator that works with KFX format.)

The FONT_RAMP feature for setting custom font sizes on e-ink Kindle devices does not work with KFX format. However, user loaded custom fonts do work for KFX on devices that support that feature.

Books converted by the plugin may not function properly in all devices and firmware versions. KFX has been extended over time. Amazon checks capabilities and only delivers KFX format if the device supports all of the features used within the book. Sideloading bypasses this check and may lead to incorrect book rendering. Because of this, the use of the most recent firmware and app versions is recommended for reading books in KFX format.

Firmware version 5.6.5 or later is required for Kindle devices. Some e-book features, such as HTML tables and Page Flip, require firmware 5.7.2 or later. Firmware 5.8.5 or later is required for manga. Arabic language books require firmware 5.9.6.1 or later.

It is unknown whether or not the Kindle Previewer can be made to function under Linux/Wine. This plugin does not currently support that environment.

This plugin requires calibre version 2.0 or later. Calibre version 2.80 or later is required for the automatic sideloading of KFX format to Kindle devices and viewing of metadata for sideloaded KFX files. Calibre version 3.29 or later is required to create cover thumbnails when KFX books are sent to a Kindle device.

The Metadata Writer can replace an existing cover in a KFX book, but cannot create a new cover in a book that does not already have one.

Version History:

Spoiler:

Version 1.28.0 - 02 Mar 2019

Add metadata (asset_id) required to allow Vocabulary Builder to work with sideloaded KFX books.

Preserve publication date metadata during conversion.

Detect Previewer versions up to 3.29.2.

Version 1.27.0 - 08 Feb 2019

Support Dutch language books. (Fixes "Unknown com.amazon.yjconversion feature: reflow-language-2")

Detect Previewer versions up to 3.29.1 and remove support for Previewer versions older than 3.27. (Newer Previewer versions produce better results.)

Allow the ".last" CLI function to work with Previewer version 3.29 and later.

Version 1.26.0 - 20 Dec 2018

Fix failure converting the most recent book from the Kindle Previewer GUI using the command line interface when file names containing non-ASCII characters are present in the user's TEMP directory.

Remove display:block styling added to <br> elements by calibre during conversion since it causes incorrect results in the Kindle Previewer.

Version 1.25.0 - 08 Dec 2018

Remove "data-AmznRemoved" and "data-AmznRemoved-M8" attributes sometimes left over in books obtained from Kindle sources.

Add "kfx_output_ignore_asin_metadata" plugin tweak that when to set to True will cause the plugin to always generate a new unique ASIN even if an ASIN is present in the book's identifiers or internal metadata. (This can be used to prevent a Kindle from downloading cover thumbnails that correspond to an ASIN that may happen to be present in the book's metadata, allowing a thumbnail set by calibre to remain unchanged.)

Detect Previewer versions up to 3.28.1. (Newer Previewer versions produce better results.)

Version 1.24.0 - 11 Nov 2018

Detect Previewer versions up to 3.28 and remove support for Previewer versions older than 3.24. (Newer Previewer versions produce better results.)

Support additional KFX features. (reflow-style-11)

Version 1.23.0 - 11 Oct 2018

Detect Previewer versions up to 3.27 and remove support for Previewer versions older than 3.23. (Newer Previewer versions produce better results.)

Support additional KFX features.

Version 1.22.0 - 13 Aug 2018

Detect Previewer versions up to 3.25 and remove support for Previewer versions older than 3.22. (Newer Previewer versions produce better results.)

The default conversion behavior has been changed to create a book (EBOK) instead of a personal document (PDOC). Removed the conversion option "Always create book instead of personal document" and added "Create personal document instead of book". Removed the command line option "--book" and added "--doc".

Fix conversion warnings for some missing files in the source EPUB not always being reported in the conversion log. (Some missing files may be reported multiple times as a side effect of this change.)

Fix incompatibility with calibre versions before 3.0 introduced in plugin version 1.17.0. (Fixes "AttributeError: 'apsw.Cursor' object has no attribute 'fetchone'")

Version 1.21.1 - 20 Jul 2018

Prevent conversion failure when a book contains an unexpected symbol count.

Version 1.21.0 - 20 Jul 2018

Detect Previewer versions up to 3.24. (Newer Previewer versions produce better results.)

Additional fixes applied to the source EPUB before conversion to work around the Previewer not handling URL escape in links. This mostly affects books whose internal file names contain spaces.

Extract additional conversion guidance from Previewer conversion logs.

Truncate the guidance report if it contains an excessive number of entries.

Version 1.20.0 - 25 Jun 2018

Add "kfx_output_previewer_path" plugin tweak to set the location where Kindle Previewer has been installed in cases where the default location was not used.

Detect Previewer versions up to 3.23 and remove support for Previewer versions older than 3.21. (Newer Previewer versions produce better results.)

Modify source file fix ups that are used when Kindle Previewer 3.23 or later is installed.

Version 1.19.0 - 21 May 2018

Allow conversion of comics with Guided View produced using Kindle Create version 1.6 in Early Access mode under Windows.

Fix error when converting KPF files produced by Kindle Create version 1.6: "CorruptError: malformed database schema (?)"

Improve location number generation.

Fix cover image metadata not being set correctly for comics.

Fix page numbers in a NAV page-list were being ignored if landmarks were also present.

Allow the ".last" CLI function to work with recent Previewer versions. (Fixes error: "No Kindle Previewer KDF database file found. No conversion results are available.")

Detect Previewer versions up to 3.22.

Version 1.18.0 - 09 Apr 2018

Set cover thumbnail to first page when converting from fixed layout KPF to KFX.

Version 1.17.0 - 02 Apr 2018

Detect Previewer versions up to 3.21.

Remove support for Previewer versions older than 3.17. (Newer Previewer versions produce better results.)

Internal restructuring and performance improvements.

Version 1.16.0 - 20 Feb 2018

Add support for Kindle Previewer version 3.20. (Fixes conversion error message detection.)

Allow repackaging of KFX-ZIP files into KFX containers using the command line interface. This works only for files that do not contain DRM.

Improve detection of unsupported KFX features.

Version 1.15.0 - 11 Jan 2018

Add support for Kindle Previewer version 3.17.1. (Fixes "Kindle Previewer error: Exception in thread "main" com.b.a.n: Unknown option: -allowYJConversionForJP")

Version 1.14.2 - 08 Jan 2018

Fix bug introduced in version 1.14.0. The ASIN value from book identifiers is not being used during conversion.

Version 1.14.1 - 05 Jan 2018

Fix error when updating to version 1.14.0: ImportError: cannot import name AUTO_PAGES

Version 1.14.0 - 05 Jan 2018

Allow the lookup name for a custom column holding the desired number of pages for each book to be specified when creating approximate page numbers.

Improve support for Kindle Previewer version 3.17. Handle change to the format of conversion guidance. Allow conversion of books containing MathML.

Remove support for Previewer versions older than 3.15. (Newer Previewer versions produce better results.)

Retry conversion with location map generation disabled if the first conversion fails with an accuracy threshold error. This allows more books to be successfully converted. (Previewer 3.17 or later is required.)

Additional fixes applied to the source EPUB before conversion.

Prevent plugin installation under Linux since Kindle Previewer 3 does not run under that OS.

Version 1.13.0 - 22 Nov 2017

Allow the optional creation of approximate page numbers in cases where real page numbers are not present in the source file of the book.

Added a Metadata Writer Plugin to allow the metadata and cover of existing KFX books to be updated.

Allow an EPUB file that has been cleaned of some potential problems to be saved using the CLI.

Fix occasional non-working links in the table of contents of books converted from AZW3 format.

Possible fix for reported conversion error: "Kindle Previewer error: Error(YJEpubAdapter):E00601: Internal error occured. java.lang.UnsatisfiedLinkError: ...\\shared.dll: Can't find dependent libraries"

Version 1.12.1 - 02 Nov 2017

Attempt to prevent error in Preferences -> Output Options.

Version 1.12.0 - 18 Oct 2017

Show conversion guidance produced by the Kindle Previewer and allow conversion of Arabic and Japanese language books when running Mac OS. (Accidentally left out of previous release.)

Added error logging to help detect the cause of Kindle Previewer failures.

Version 1.11.0 - 16 Oct 2017

Change the ".last" CLI function to work with recent Previewer versions. (Fixes error: "No Kindle Previewer KDF database file found. No conversion results are available.")

Remove the "kfx_output_force_cde_type_ebok" plugin tweak. This feature is now supported through plugin configuration.

List features and metadata of the KFX book in the conversion job log.

Show conversion guidance produced by the Kindle Previewer in the conversion job log. (Previewer version 3.15 or later is required.)

The full conversion log from the Kindle Previewer will no longer be included in the conversion job log. This can be overridden through plugin configuration.

Allow Arabic language books to be converted. (Previewer version 3.14 or later is required. Support for Arabic in Kindle apps/devices is currently incomplete.)

Allow Japanese language books to be converted. (Previewer version 3.15 or later is required. Only horizontal text is supported in Previewer 3.15.)

Handle some additional EPUB problems that would otherwise cause the Previewer to fail. (order of language declarations, incorrectly coded horizontal ellipsis entity)

Version 1.10.0 - 30 Aug 2017

Adjustments to output format to better match KFX files currently produced by Amazon. (Reading these files with the latest Kindle firmware/app version is recommended for best results.)

Fix occasional error during conversion: "TypeError: info() takes exactly 2 arguments (3 given)"

Attempt to work around occasional error during conversion caused by unexpected data produced by the Previewer: "yj.eidhash_eid_section_map has extra: ..."

Version 1.9.0 - 09 Aug 2017

Remove support for Previewer versions older than 3.13.

Allow English language MS Word DOC and DOCX files and MOBI files (created using kindlegen version 2.9 without the -dont_append_source option) to be converted to KFX using the Command Line Interface.

Allow Chinese language books to be converted.

Improve performance when converting large and complex books.

Improve reporting of conversion errors. Do not truncate (tail) Previewer conversion logs in the calibre conversion job log.

Handle more EPUB problems that would otherwise cause the Previewer to fail. (badly sized transparent GIFs, "onload" attribute)

Further internal changes based on the Amazon Ion specifications.

Version 1.8.1 - 31 Jul 2017

Remove the conversion time limit set by the plugin so that larger books can be converted. (A time limit can sill be set in calibre using Preferences, Miscellaneous, Abort jobs that take more than...)

Version 1.8.0 - 12 May 2017

Allow conversion of fixed layout comics and manga using Kindle Previewer version 3.10.1 or higher using the command line interface. See "Converting Comics and Manga" in the first post of this thread for more information.

Allow conversion of KPF files produced by the Amazon Kindle Create software using the command line interface.

Allow conversion of unzipped EPUBs in the command line interface by using an OPF as the input file.

Allow conversion of page numbers contained in an EPUB-3 NAV page-list within the source EPUB.

Include Previewer conversion logs in the calibre conversion job log even when conversion is successful.

Attempt to correct failures for some users when running 64-bit calibre under Windows Vista.

Internal changes to better match the Amazon Ion specifications. (KFX is based on Ion.)

Version 1.7.1 - 26 Mar 2017

Fix bug when running under MacOS introduced in version 1.7.0. (Fixes "TypeError: encode() argument 1 must be string, not None")

Version 1.7.0 - 22 Mar 2017

Fix incompatibility with Kindle Previewer 3.9 beta and improve handling of conversion error logs. (Fixes "KFX creation error: Unexpected conversion log: log_processEpub.txt")

Improve handling of non-ASCII system environments. (Untested under Mac OS.)

Remove support for Previewer versions older than 3.7 beta.

Minor changes and bug fixes.

Version 1.6.0 - 23 Dec 2016

Add support for Kindle Previewer 3.7 beta. (Previewer version 3.7 is recommended for use with this version of the plugin. It converts some books with unusual formatting that previous versions could not.)

Fix bug that caused some problems with the NCX TOC to not be corrected before conversion.

Version 1.5.3 - 12 Dec 2016

Fix conversion when using Kindle Previewer 3.6 beta under Mac OS.

Version 1.5.0 - 27 Nov 2016

Add support for Kindle Previewer 3.6 beta. (That is now the recommended version of the Previewer for use with this version of the plugin. It converts some books containing tables that previous versions could not.)

Capture log files during the first conversion attempt. This avoids the need to perform a second conversion to capture logs if the first one fails.

Automatically retry after conversion errors that occur inconsistently.

Version 1.4.0 - 24 Oct 2016

Allow a "mobi-asin" identifier to be used as the source of the book's ASIN.

The plugin tweak "kfx_output_force_cde_type_ebok = True" will cause converted books to no longer be marked as personal documents.

Version 1.3.0 - 20 Sep 2016

An invisible character will be added to empty table cells to work around a Previewer bug. (A visible character was used previously.)

If no ASIN is specified the converted book will be marked as a "personal document" and a unique ASIN will be generated. (An ASIN is required for the support of cover thumbnails on e-ink Kindles.)

Add support for Kindle Previewer 3.5 beta. (KFX generation from Previewer version 3.5 is unchanged from version 3.4.)

Version 1.2.0 - 15 Aug 2016

Add support for Kindle Previewer 3.4 beta. (That is now the recommended version of the Previewer for use with this version of the plugin.)

Allow conversion of page numbers contained in an NCX pageList or page-map.xml within the source EPUB. (Page numbers must be present in the EPUB. "Fake" page numbers will not be produced. EPUB 3 page-list and Amazon APNX are not supported.)

Correct some issues in the source EPUB that could prevent conversion by Previewer: character encoding, obfuscated fonts, background RECT in SVG, language "UND", missing/duplicated title, table cell with no visible text, and a few others.

Include the book title in pop-up conversion failure dialogs.

Save the Kindle Previewer version used within created KFX files to aid in future debugging.

Generate a unique content_id for books that do not have an ASIN.

Version 1.1.2 - 08 Apr 2016

Fix inability to highlight text in converted books on Kindle devices.

Assure author names are in the correct format for sorting.

Allow conversion using the plugin within calibre under Mac OS.

Version 1.1.1 - 30 Mar 2016

Fix error from calibre when the plugin is updated from an older version.

Version 1.1.0 - 30 Mar 2016

Add command line interface for advanced users. This provides a means for Mac users to produce KFX files since conversion within calibre is not currently working under Mac OS.

Version 1.0.0 - 26 Mar 2016

Initial release.

[jhowell]

Open KFX with Kindle Previewer 3

KFX files produced by the KFX Conversion Output plugin or imported using the KFX Input plugin can be viewed using Kindle Previewer 3. However, the Previewer supports the file extension .azw8 for opening KFX files instead of .kfx. In order to open a KFX file with the Previewer it must first be copied or renamed so that it has a .azw8 extension.

The "Open KFX with Kindle Previewer 3.bat" script can be installed in Windows to allow files with a .kfx extension to be viewed more easily.

Installation:

• Kindle Previewer 3 must already be installed for the current user.
• Download the .zip file attached to this post.
• Extract the enclosed "Open KFX with Kindle Previewer 3.bat" file.
• Place this .bat file in a permanent location of your choice on the PC.
• Right-click the .bat file and select "Run as administrator". The script will report success or failure. Press enter to close it. (This will associate the script with .kfx files. Repeating this step will remove the association.)

Once associated the script file will execute whenever .kfx files are to be opened.


Usage:

Once installed you can then select a converted book in calibre and in the "Formats" section of the book details panel and simply click on "KFX" to open it with the Previewer. The .kfx file will be copied to a temporary .azw8 file and opened with Kindle Preview 3.


Limitations:

This script uses a simple method to create the file type association, not the best possible method.

A Command window will briefly pop up whenever the script is activated.

It has only been tested under Windows 10.

It is not available for MacOS or Linux.

Version History:

Spoiler:

Version 1.0.0 - 24 Oct 2016

Initial release.

[cerem0ny]

Thank you! You're a genius and inspire me to search and regain faith and trust in humanity. God bless!

...

but what IS funny is that the okugin seems to have more success converting .kfx than the previewer itself! Go figure. Why is that?!?

[jhowell]

Originally Posted by cerem0ny

but what IS funny is that the okugin seems to have more success converting .kfx than the previewer itself! Go figure. Why is that?!?

The first step in the conversion process is to convert the source format to EPUB, even if the book is already in EPUB format. This process can fix some things that would otherwise cause the Previewer to fail.

For example, the Previewer fails if the source EPUB does not have UTF-8 character encoding. The initial conversion step automatically converts other encodings to UTF-8.

[Nate the great]

Here's the log from my failed attempt:

[jhowell]

Originally Posted by Nate the great

Here's the log from my failed attempt: ...

There isn't much I can tell from it. The Previewer's conversion routine, which runs under a Java installation included with the Previewer, somehow failed to launch. There is nothing in the log to indicate why. I can't tell if the problem is specific to Windows 7 or your particular PC.

I'll try to think of a way to have the plugin gather more information on this type of failure. In the mean time I have some questions to help diagnose the problem:

• Were there any other error messages displayed on your system when the conversion failed? If so, what?
  
• Have you used the Kindle Previewer to verify that it functions properly on your system? You could try opening this book and verifying that it displays with Enhanced Typesetting (hyphenation) in the Previewer. Press Ctrl-I after opening the book in the Previewer and make sure that the information dialog shows "Enhanced Typesetting: Supported".
  
• If you are running an anti-virus program could you try disabling it before launching a conversion to see if that helps?

[amordechai]

Hi. I registered only for this.

First: thank you SO much for doing this!

Sadly I tried using your plugin with the latest version of calibre and Kindle previewer 3 on my mac but i got an error message and the conversion failed.
here is the log: http://pastebin.com/uUhqQWny
I'm not using any antivirus that could interfere.

It crashed with the Frankenstein book you linked. Later I tried to import the book in kindle previewer and got the "Enhanced Typesetting: Supported" message, so this seems to work.

Thank you again for your work!
Antonio

[jhowell]

Originally Posted by amordechai

Sadly I tried using your plugin with the latest version of calibre and Kindle previewer 3 on my mac but i got an error message and the conversion failed.

I was not able to test the plugin under Mac OS so it is not surprising that it failed to work.

I had to make some guesses about where files are located in the Mac version of the Previewer and I guessed wrong. I will have a new version of the plugin ready in a day or two that may fix the problem. Please post a new log if the new version of the plugin still fails for you.

Thanks for your error report.

[Nate the great]

Originally Posted by jhowell

There isn't much I can tell from it. The Previewer's conversion routine, which runs under a Java installation included with the Previewer, somehow failed to launch. There is nothing in the log to indicate why. I can't tell if the problem is specific to Windows 7 or your particular PC.

I'll try to think of a way to have the plugin gather more information on this type of failure. In the mean time I have some questions to help diagnose the problem:

• Were there any other error messages displayed on your system when the conversion failed? If so, what?
  
• Have you used the Kindle Previewer to verify that it functions properly on your system? You could try opening this book and verifying that it displays with Enhanced Typesetting (hyphenation) in the Previewer. Press Ctrl-I after opening the book in the Previewer and make sure that the information dialog shows "Enhanced Typesetting: Supported".
  
• If you are running an anti-virus program could you try disabling it before launching a conversion to see if that helps?

The converter works, yes, and it did make a couple ebooks with the enhancements. I saw the hyphenation.

Then it asked me to update to 3.1, which I did.

After that I ran the plugin again in calibre, and this time it told me that KindleImageConverter failed (twice, on one job).

And then, on the next try, it converted the ebook which had failed 4 times already.

So now it is working - sorta.

[Alice Green]

How can I use the Amazon Kindle Previewer to convert from EPUB to KDF, when I open the EPUB via Kindle Previewer, it convert it to Mobi format. I have not seen any convert options to KDF.

[Nate the great]

Originally Posted by Alice Green

How can I use the Amazon Kindle Previewer to convert from EPUB to KDF, when I open the EPUB via Kindle Previewer, it convert it to Mobi format. I have not seen any convert options to KDF.

The easiest way is to do it via calibre.

But to be clear, that "Mobi" file you just made isn't just a Mobi file. Did you notice how it is several times the size of the Epub you started with? That's because it actually contains several different ebooks, including the original Epub, Mobi, and the new KFX.

[amordechai]

Originally Posted by jhowell

I was not able to test the plugin under Mac OS so it is not surprising that it failed to work.

I had to make some guesses about where files are located in the Mac version of the Previewer and I guessed wrong. I will have a new version of the plugin ready in a day or two that may fix the problem. Please post a new log if the new version of the plugin still fails for you.

Thanks for your error report.

I wil definitely try the new version of the plugin and report back.

Thanks for your work!

[cerem0ny]

Wish I kept my Aura H2O! Can't remember any problems with the typography but this conversion process is so hit and miss, it's annoying! Props for the plugin however

[enjambement]

if you use the book "Azw3", you can convert the book in KFX always. Sorry for my English, I'm Italian.

[jhowell]

Originally Posted by Nate the great

And then, on the next try, it converted the ebook which had failed 4 times already.
So now it is working - sorta.

Conversion by the plugin is dependent on Amazon's Kindle Previewer working properly and Amazon didn't do a very good job of with that software. For me it fails on more books than it works, but it does work or fail consistently on any particular book.

I don't know why it functions inconsistently for you and presumably others. It may be due to a bug in the plugin, but since I can't duplicate the problem I am unable to diagnose it further at the moment.

Originally Posted by Alice Green

How can I use the Amazon Kindle Previewer to convert from EPUB to KDF, when I open the EPUB via Kindle Previewer, it convert it to Mobi format. I have not seen any convert options to KDF.

You do not need to locate the KDF files created by the Kindle Previewer to make use of the KFX Output plugin. It invokes the conversion function of the Previewer and uses the resulting KDF file automatically. But for those interested, I will describe more of how it works.

After opening a book with the Kindle Previewer you will usually see a dialog box with the message "Your book successfully converted into Kindle format" with a link to a folder containing all of the MOBI files produced by the Previewer. This message is a bit misleading since it doesn't let you know whether or not allowing Enhanced Typesetting succeeded for that book.

The Previewer generates a KDF file for each book, but only if it succeeds at converting it for Enhanced Typesetting. To tell if this was successful you need to check the book information screen using Ctrl-I or by selecting View->Book Information via the program menu. If this shows "Enhanced Typesetting: Supported" then a KDF file was produced. If it show "Not Supported" then it failed to produced the KDF.

The KDF file will be in a recently created subdirectory of your TEMP directory. A different random subdirectory is used for each book. You need to locate the one created at the time that you opened your book in the Previewer. For example, on my Windows PC the TEMP directory is "C:\Users\John\AppData\Local\Temp" and I can find the most recently converted book within it named "145a-0754-be2f-1d6e\book\book.kdf".