[Conversion Output] KFX Output

[oops]

Originally Posted by alessandrokindle

Hi, I have an EPUB file that contains different types of fonts. After using your tool, the fonts are not preserved...

Hi. I ran into the same problem and have root-caused it and have a fix. First of all, the KFX Output plugin is not at fault. Kindle Previewer/KindleGen are the ones that are misbehaving. In my case, there would be messages in "KPR.Log" where the font files would be invalidated. The reason is that "FontValidator.exe" is a) a 32-bit application and b) requires the x86 2013 VC redistributable DLLs. It might complain only about "msvcr120.dll" and you'll be tempted to just copy it alongside the executable, but looking at the executable in Dependency Walker, it needs its supporting libraries as well. In my case the redistributable was not installed--I don't play games on this machine and Steam tends to be the one installing these nowadays. Installing the redistributable fixed the problem.

For reference here is what the error looks like in "KPR.Log":

Spoiler:

[ERROR] Invalidating the font:%LOCALAPPDATA%\Temp\2f029474-7d9d-47a1-b485-786b93621966\conv_out\book\res/rsrc55B

In addition, some fonts might require conversion to TrueType glyphs to display properly, so make sure that you take care of that in case it is still not working for you. This is what that warning from KindleGen looks like; note that it doesn't bubble up to Kindle Previewer's conversion log:

Spoiler:

************************************************** ***********
Amazon kindlegen(Windows) V2.9 build 0000-kdevbld
A command line e-book compiler
Copyright Amazon.com and its Affiliates 2015
************************************************** ***********
...
Warning(prcfile):W14029: CFF/Type1 (Postscript) embedded font included in your source may not render clearly on all Kindle readers. Please refer to Kindle Publishing Guidelines on the recommended usage of embedded fonts. Font File : ...

Originally Posted by jhowell

If that doesn't work I would need to see an example of an EPUB the loses its fonts when converted in order to diagnose this further.

Hopefully @jhowell, this saves you some sleuthing.

[jhowell]

Originally Posted by oops

Hi. I ran into the same problem and have root-caused it and have a fix.

Thanks for the useful information.

[alessandrokindle]

Originally Posted by oops

Hi. I ran into the same problem and have root-caused it and have a fix. First of all, the KFX Output plugin is not at fault. Kindle Previewer/KindleGen are the ones that are misbehaving. In my case, there would be messages in "KPR.Log" where the font files would be invalidated. The reason is that "FontValidator.exe" is a) a 32-bit application and b) requires the x86 2013 VC redistributable DLLs. It might complain only about "msvcr120.dll" and you'll be tempted to just copy it alongside the executable, but looking at the executable in Dependency Walker, it needs its supporting libraries as well. In my case the redistributable was not installed--I don't play games on this machine and Steam tends to be the one installing these nowadays. Installing the redistributable fixed the problem.

For reference here is what the error looks like in "KPR.Log":

Spoiler:

[ERROR] Invalidating the font:%LOCALAPPDATA%\Temp\2f029474-7d9d-47a1-b485-786b93621966\conv_out\book\res/rsrc55B

In addition, some fonts might require conversion to TrueType glyphs to display properly, so make sure that you take care of that in case it is still not working for you. This is what that warning from KindleGen looks like; note that it doesn't bubble up to Kindle Previewer's conversion log:

Spoiler:

************************************************** ***********
Amazon kindlegen(Windows) V2.9 build 0000-kdevbld
A command line e-book compiler
Copyright Amazon.com and its Affiliates 2015
************************************************** ***********
...
Warning(prcfile):W14029: CFF/Type1 (Postscript) embedded font included in your source may not render clearly on all Kindle readers. Please refer to Kindle Publishing Guidelines on the recommended usage of embedded fonts. Font File : ...

Hopefully @jhowell, this saves you some sleuthing.

This fixed the problem, thank you so much!

[tomsem]

Originally Posted by alessandrokindle

This fixed the problem, thank you so much!

I take it that this is a Windows-specific issue?

[tomsem]

I've been using CLI to convert KPF (from Kindle Create) to KFX, but just tried using Convert in calibre after importing the KPF. The job ran for a couple of minutes without progressing beyond 67%, so I killed it (the CLI does this in a couple of seconds).

I assume this workflow is not supported. Is it possible to support it?

[jhowell]

Originally Posted by tomsem

I've been using CLI to convert KPF (from Kindle Create) to KFX, but just tried using Convert in calibre after importing the KPF. The job ran for a couple of minutes without progressing beyond 67%, so I killed it (the CLI does this in a couple of seconds).

I assume this workflow is not supported. Is it possible to support it?

calibre's conversion workflow always converts to EPUB as an intermediate format, so in your case it will convert KPF to EPUB and then EPUB to KFX using the Kindle Previewer. Additionally calibre's conversion does not properly support fixed layout books such as comics. So for now it is best to use the CLI to convert KPF to KFX.

At some point I may add a toolbar button to this plugin to perform the equivalent conversion of the CLI within the calibre GUI, bypassing calibre's conversion pipeline. But that won't be coming in the near future.

[tomsem]

Originally Posted by jhowell

calibre's conversion workflow always converts to EPUB as an intermediate format, so in your case it will convert KPF to EPUB and then EPUB to KFX using the Kindle Previewer. Additionally calibre's conversion does not properly support fixed layout books such as comics. So for now it is best to use the CLI to convert KPF to KFX.

At some point I may add a toolbar button to this plugin to perform the equivalent conversion of the CLI within the calibre GUI, bypassing calibre's conversion pipeline. But that won't be coming in the near future.

Got it, thanks!

[tomsem]

I have an interesting issue.

I've started doing Add from folders and subfolders for the first time, as I'm doing some housekeeping on my calibre library and re-doing the earliest ones added when I didn't know as much - and also I prefer KFX to AZW3 for side loading and when these are fixed content, the conversion to ePub with From KFX is more of a no brainer (in past I had to remember to use KindleUnpack's conversion).

I'm downloading with Windows app, pointing it at the My Kindle Content folder and selecting the aforementioned option. It asks if I want to combine books in folders as one book, or treat them as separate.

If I chose to 'combine' it creates books with goofy titles (ASIN?) that end with .azw extension and inside that are the imported book in KFX, usually. And that goofy title is used for the book file (.kfx) and another with the same name with .MD extension.

These .MD files are in binary format, no idea where they came from but it was not in the source folder, and .MD is supposed to be plain text.

I thought I could exclude .MD from calibre's supported file preference but it had no effect, and even if there was no .MD file the title/filename is not as expected.

This instance of calibre is only used to de-drm books, then I import them to my Mac via cloud folder. But Saving to this folder propagates the goofy book name, making it hard to merge with the previous book record in order to update with the refreshed content.

The workaround is to treat them as separate books. Then the MD's have the goofy names and the KFX (AZW3 or MOBI) have the expected names. I can delete the MD entries, select the books and Save to the transfer folder.

I don't know where these MD files are coming from (it's not in the source folder) but I believe it is only there with KFX imports so that's why I'm posting here.

[jhowell]

Originally Posted by tomsem

If I chose to 'combine' it creates books with goofy titles (ASIN?) that end with .azw extension and inside that are the imported book in KFX, usually. And that goofy title is used for the book file (.kfx) and another with the same name with .MD extension.

The combine option tells calibre to treat each of the files in the folder as a separate instance of the same book. Books in KFX format are composed of multiple files that together make up only one instance of the book.

The Kindle for PC app uses the file extension .md for KFX metadata files. Calibre will interpret that incorrectly as being a separate copy of the book in markdown format.

See the section "Importing from Kindle for PC" under "Importing KFX books into calibre" in the first post of the KFX Input plugin thread for information on how to ignore those .md files and make the import work better.

Originally Posted by tomsem

These .MD files are in binary format, no idea where they came from but it was not in the source folder, and .MD is supposed to be plain text.

They are there in the folder that makes up a book in KFX format along with .azw, .res, and .voucher files. Your screenshot shows that you have file extensions hidden in Windows file explorer. (That is the default unless changed.)

[rupor]

Hello jhowell

Thank you for your wonderful work - it is really inspiring!

I have a technical question, not sure this is the right place to ask, but I will try.

I am converting epub to kfx using Calibre with your KFX Output plugin. When I am doing it on Windows (calibre-portable, gui, convert individual, save to disk) I got output in which text-indent on the paragraph style is always expressed in "%" ($314). If I am using yshalsager/calibre-with-kfx (which basically runs KP3 on Wine) text-indent is in "em" ($308). I am pretty sure that versions of your plugins are the same in both cases and version of KP3 is pretty close if not the same too. I spent quite a bit of time trying to trace it down through Calibre code and such, but was not able to so far.

Reason for me asking is simple - I am doing direct conversion of FB2 books into KFX (using my own code) and one of the users noticed that when scaling up font size on the device in one case indentation of first line of paragraph scales with the font and in another it does not (obvious, right?).

All of that is pretty easily reproducible in KP3 GUI, but what makes it even more puzzling is the fact, that if I open source epub in KP3 directly - text-indent scales! When I open KFX file produced by abovementioned yshalsager/calibre-with-kfx - it scales. But when I open KFX file produced by Calibre and KFX Output plugin (same computer, same KP3, calibre-portable) - it does not.

In all cases the only enabled Calibre plugins are KFXInput and KFXOutput.

So I was wondering if you may have any idea where to look? I am extremely curious what may cause such behavior.

Thank you!

[jhowell]

Originally Posted by rupor

I am converting epub to kfx using Calibre with your KFX Output plugin. When I am doing it on Windows (calibre-portable, gui, convert individual, save to disk) I got output in which text-indent on the paragraph style is always expressed in "%" ($314). If I am using yshalsager/calibre-with-kfx (which basically runs KP3 on Wine) text-indent is in "em" ($308). I am pretty sure that versions of your plugins are the same in both cases and version of KP3 is pretty close if not the same too. I spent quite a bit of time trying to trace it down through Calibre code and such, but was not able to so far.

When converting to KPF/KFX the Kindle Previewer tries to normalize some of the book's formatting for consistent look on Kindles. It uses heuristics to determine whether some units should be changed to scale with the screen size (%) or font size (em) based on how the book is formatted. "em" units are used for text-indent in the cases I have examined. Something about the specific formatting or conversion process in your books might be triggering something different to occur.

Originally Posted by rupor

All of that is pretty easily reproducible in KP3 GUI, but what makes it even more puzzling is the fact, that if I open source epub in KP3 directly - text-indent scales! When I open KFX file produced by abovementioned yshalsager/calibre-with-kfx - it scales. But when I open KFX file produced by Calibre and KFX Output plugin (same computer, same KP3, calibre-portable) - it does not.

It seems you are using the calibre GUI for conversion while calibre-with-kfx uses the KFX Output plugin's CLI. One possible source of different results is that the conversion process in calibre's GUI performs additional steps that can alter a book's formatting prior to passing it to KFX Output.

Originally Posted by rupor

So I was wondering if you may have any idea where to look? I am extremely curious what may cause such behavior.

You could try using the CLI of KFX Output in order to avoid the calibre conversion pipeline and more closely match what calibre-with-kfx is doing.

[rupor]

Thank you for the answers.

calibre-with-kfx is running "calibre-debug -r "KFX Output" -- <infile> [<outfile>] [options]" - is that what you mean by "KFX Output CLI"?

The part where Kindle Previewer does its magic I investigated pretty thoroughly before switching my focus to what Calibre does (or does not do)

Just trying to understand the behavior - I do not think KFX Output converts those units and I am quite certain that KPF/KDF output from Kindle Previewer contains "em" units in some cases. However when I got resulting KFX from Calibre GUI conversion - all units are "%"...

[jhowell]

Originally Posted by rupor

calibre-with-kfx is running "calibre-debug -r "KFX Output" -- <infile> [<outfile>] [options]" - is that what you mean by "KFX Output CLI"?

Yes. It is documented in the first post of this thread.

Originally Posted by rupor

The part where Kindle Previewer does its magic I investigated pretty thoroughly before switching my focus to what Calibre does (or does not do)

If I am reading that correctly, your conclusion is that the Kindle Previewer always produces text-indent in percent for reflowable books. That is incorrect.

Originally Posted by rupor

Just trying to understand the behavior - I do not think KFX Output converts those units and I am quite certain that KPF/KDF output from Kindle Previewer contains "em" units in some cases. However when I got resulting KFX from Calibre GUI conversion - all units are "%"...

I did some further testing and I find that text-indent in em is preserved by the Kindle Previewer as long as no left margin is set. The presence of a non-zero left margin results in text-indent being changed to a percent value.

This can be a problem if the calibre GUI conversion pipeline is used since the margins specified in the "Page setup" tab all default to 5pt. Changing the left margin setting there to either "0" or "Unchanged" prior to conversion should result in em units being preserved for text-indent.

I will make a note of this finding in the KFX Output plugin documentation and see whether I can do anything to mitigate this behavior in the future.

[rupor]

Originally Posted by jhowell

I did some further testing and I find that text-indent in em is preserved by the Kindle Previewer as long as no left margin is set. The presence of a non-zero left margin results in text-indent being changed to a percent value.

Who would've guessed such a thing! This puts my mind at ease somewhat, it should be save for my code to output "em" unconverted for all horizontal margins and text-indent then.

Thanks to your pointer I found missing piece now and updated my doc, here is what I missed before: It is converted to "%" if text layout is normal and for original unit being "em" and if

- Text-indent value exceeds 4.0em, OR
- margin-left is non-zero, OR
- padding-left is non-zero

I am not sure I need to repeat all of this, may be some day. Kind of wonder why such complexity is needed at all...

Thank You for your help - all is clear now!

[njpig]

Hello,

This epub file sent to kindle via https://www.amazon.com/sendtokindle is not converted to kfx format. Anybody has clue on this?

Thanks, Fei.