KFX output linux

[Ruskie_it]

Originally Posted by jhowell

The plugin is invoking the Previewer via the CLI. I do not know what is causing the attempts to open a window. It is clearly not running very well under Wine on your system.

Indeed... Is there any place where I could find more detailed information about the crash?

[jhowell]

Originally Posted by Ruskie_it

Indeed... Is there any place where I could find more detailed information about the crash?

The log you posted shows shows the Previewer running to completion but reporting an unknown conversion error. Conversion is done by invoking multiple subprocesses. One of them is likely failing in an unexpected way. I have no clue how to even begin debugging that under Linux and Wine.

My only suggestion is to try using an earlier version of the Kindle Previewer, such as 3.60 released November 2021. That was reported to work under Linux/Wine. But even that might not succeed since it was done using specific Linux and Wine releases. You can search the KFX Output plugin thread and see multiple reports of success and failure under Linux using different software versions.

Unfortunately Amazon does not make older versions of their software available. You may be able to find them somewhere on the web with the risk of obtaining a copy containing malware.

[Ruskie_it]

Originally Posted by jhowell

The log you posted shows shows the Previewer running to completion but reporting an unknown conversion error. Conversion is done by invoking multiple subprocesses. One of them is likely failing in an unexpected way. I have no clue how to even begin debugging that under Linux and Wine.

Alright, that has been fun as long it lasted.
Thanks!

[xxyzz]

Is it doable to write a program to replace the kindle previewer cli that just have some basic features?

[jhowell]

Originally Posted by xxyzz

Is it doable to write a program to replace the kindle previewer cli that just have some basic features?

It is certainly possible to do something very limited but that would probably just provoke requests to add functionality. It would take more work than I am willing to put in on a project that has no use or interest for me.

Perhaps someone with a greater knowledge of Linux and Wine might be able to figure out why the Previewer CLI is failing and come up with a work around, such as using an earlier version of it.

[xxyzz]

Wine probably need some configurations to work correctly, maybe dependencies are missing? This document might be helpful if someone want to fix the wine error: https://wiki.archlinux.org/title/Wine

I'd like to create a simple implementation of the preview cli. I want to find the KFX file format in your KFX Input plugin code, but don't know where to look at, could you please give some hints?

[Quoth]

Look at Winetricks and Playonlinux on your Linux. Both (and other WINE) stuff are on Mint's Software Manager GUI.

I used to always setup WINE, but I've lost interest in it. It's most use for 32 bit Windows programs that don't work on 64 bit Win7 or Win10, but you need to set an env variable to get that otherwise there is only 64 bit support.
I do have an old LT Spice, Word & Excel 2003, Orbitron, VB6 based uTracer and an old Eagle CAD I had on XP (all 32 bit) and a few other 32 bit XP programs that are orphaned or developer is dead. I have also XP. Win7 and Win1o VMs on Vbox. My Weather station GUI on the XP VM, though there is a Linux (web) server program for it. I gave up on Amazon's Windows software on WINE years ago when the PC reader app stopped logging in. I think I tried Kindle Creator or Previewer years ago and they didn't work on WINE. Since we use Calibre for years now rather than Mobipocket Creator (over 12 years ago?) and test ebooks on real ereaders and switched from docx to epub for upload to Amazon Publishing, I can't see the point of the Amazon Windows SW. The Previewer isn't accurate.
Also if I want electronic to match paper I use PDF. Otherwise I prefer epub to kepub and azw3 to kfx. AFAIK the only real way to test KFX is to publish and buy the ebook via Whispernet on a suitable real Kindle. Amazon is an increasingly messed up "walled garden".
So now I only read epubs on epub compatible (usually Kobo Sage, but also a 5″ Sony and 4.7″ Chinese thing) and only use Kindles to test new layout/format designs in an ebook.

[j.p.s]

I would like to point out that xxyzz is the creator of the WordDumb calibre plugin to generate xray and wordwise files and in my opinion has excellent programming skills including designing and implementing applications that are easy (for him) to add new features in a very localized way with very few lines of code. I am amazed by the speed of his response to requests.

When he asks a question, it does not need to be second guessed or deflected. I suspect that if he receives the information he is requesting, something wonderful will result.

His code is on github if what I wrote above needs to be verified.

[jhowell]

Originally Posted by xxyzz

I'd like to create a simple implementation of the preview cli. I want to find the KFX file format in your KFX Input plugin code, but don't know where to look at, could you please give some hints?

Here is a brief overview of the significant code in kfxlib, most of which is common to the KFX Input and KFX Output plugins.

• generate_kpf_common.py, generate_kpf_using_cli.py - Invoke Kindle Previewer CLI to convert EPUB to KPF.
• ion.py - Amazon Ion format, which is the underlying data format for KFX.
• ion_binary.py - The binary representation of Ion.
• ion_symbol_table.py - Ion symbol tables.
• ion_text.py - The text representation of Ion, useful for debugging.
• kfx_container.py - KFX file structure. Ion data fragments in a compact container.
• kpf_book.py - Book structure specific to KPF format.
• kpf_container.py - KPF file structure. Ion data fragments in an SQLite database.
• resources.py - Images, fonts, and mime types.
• yj_book.py - Handle the multiple files that make up a book in KFX format. Starting point for conversion.
• yj_container.py - Book content as a list of data fragments in Ion format.
• yj_metadata.py - Book metadata.
• yj_position_location.py - The methods by which book content is indexed.
• yj_structure.py - Consistency checks for various types of Ion data fragments.
• yj_symbol_catalog.py - Dummy Ion symbol catalog.
• yj_to_epub.py - Overall conversion of KFX to EPUB.
• yj_to_epub_content.py - Conversion of book content.
• yj_to_epub_illustrated_layout.py - Conversion of Kindle in Motion books. No longer being produced.
• yj_to_epub_metadata.py - Conversion of book metadata.
• yj_to_epub_misc.py - Miscellaneous data conversion.
• yj_to_epub_navigation.py - Navigation conversion: table of contents, guide, etc.
• yj_to_epub_notebook.py - Scribe notebooks.
• yj_to_epub_properties.py - Conversion of CSS properties.
• yj_to_epub_resources.py - Conversion of fonts and images.
• yj_to_pdf.py - Conversion of KFX book images to PDF.
• yj_versions.py - Known KFX metadata.

Notes:

KFX and KPF store data structures using Amazon Ion format. That is the only documented aspect of KFX as far as I know. The data structures that make up a book are called fragments.

My code contains Ion symbols in their numeric form, such as $123. I did not include a copy of the YJ_symbols symbol table to avoid copying anything significant from Amazon. You can obtain the actual symbol table by decompiling EpubToKFXConverter-4.0.jar from Kindle Previewer and looking for the enumeration that begins with "kProperty_Invalid".

The KFX Output plugin converts KPF from Kindle Previewer to KFX. However this is mostly just a repackaging of Ion fragments. The hard work is done by the Kindle Previewer.

My code wasn't written to be especially readable. There are no comments.

[xxyzz]

I tried to use Amazon's Ion Python library but can't figure out how to read or write the container info data with a shared symbol table(SymbolTableCatalog). If I read the data I get `out of range sid` error, and dump the binary data has some extra symbol table name data.

I'm not sure if it's their code can't be used with KFX data, is this the reason you write your own ion implementation?

[jhowell]

Originally Posted by xxyzz

I tried to use Amazon's Ion Python library but can't figure out how to read or write the container info data with a shared symbol table(SymbolTableCatalog). If I read the data I get `out of range sid` error, and dump the binary data has some extra symbol table name data.

I have never tried to use Amazon implementation of Ion so I do not know how well it works for interpreting KFX data.

Originally Posted by xxyzz

I'm not sure if it's their code can't be used with KFX data, is this the reason you write your own ion implementation?

I wrote my own implementation of Ion because at that time I was unaware of Amazon's public version. Once I became aware of it I refined my implementation based on their documentation of the format rather than switching to using their version, largely as a learning experience.

[xxyzz]

I think I have figured out how to use the ion library and I'm investigating the kdf sqlite file now. And I have 4 questions so far:

- How fragment ids are created? Only section rows start from "c0", others start from a random id like the first anchor id is "a1B9".
- How eidbucket_* rows are created?
- The section rows always have an non-exists kfx_id field, for example here is the text ion data of the "c0" row in the fragments table:

$ion_1_0
section::{
  section_name: kfx_id::"c0",
  page_templates: [
    structure::{
      kfx_id: kfx_id::"i3",
      story_name: kfx_id::"l4",
      fixed_width: 1400,
      fixed_height: 2100,
      layout: scale_fit,
      float: center,
      type: container
    }
  ]
}

the "i3" fragment id doesn't exist
- The meaning of the numbers in section_position_id_map, for example, the "c117-spm" row:

$ion_1_0
section_position_id_map::{
  section_name: kfx_id::"c117",
  contains: [
    [
      1,
      kfx_id::"i11A"
    ],
    [
      2,
      kfx_id::"i11E"
    ],
    [
      3,
      kfx_id::"i11C"
    ],

I'm not sure what's 1, 2, 3.

I have attached the epub and the kdf files, I added a text ion table "text_fragments" to the db file. The book is downloaded from https://standardebooks.org.

[jhowell]

Originally Posted by xxyzz

- How fragment ids are created? Only section rows start from "c0", others start from a random id like the first anchor id is "a1B9".

The convention used for fragment id symbols has changed over time. In the end they really only need to be unique.

Conventions used in the past that can still be seen in older Amazon books include base64 values and uuids for uniqueness with varying prefixes and suffixes.

The current convention is to use a lower case letter indicating the purpose of the symbol followed by a unique value. The letter prefixes I have seen are: a=anchor, b=ruby_content, c=section, d=auxiliary_data, e=resource, f=font, i=structure, l=storyline, n=nav_container, p=path_bundle, s=style, t=kfx_id/eid, z=copy/pasted content. The suffix value increments as new symbols are created and is expressed in base-32 (0-9,A-Z excluding ILOQ).

Originally Posted by xxyzz

- How eidbucket_* rows are created?

They appear in yj.eidhash_eid_section_map fragments, present in KPF but not KFX format. The data contained is redundant with the section_position_id_map. I have not looked into why they exist or how they are created.

Originally Posted by xxyzz

- The section rows always have an non-exists kfx_id field, for example here is the text ion data of the "c0" row in the fragments table:

$ion_1_0
section::{
  section_name: kfx_id::"c0",
  page_templates: [
    structure::{
      kfx_id: kfx_id::"i3",
      story_name: kfx_id::"l4",
      fixed_width: 1400,
      fixed_height: 2100,
      layout: scale_fit,
      float: center,
      type: container
    }
  ]
}

the "i3" fragment id doesn't exist

A kfx_id is a label for a data structure, which can be an entire fragment or just a part of one. In your example "l4" is a reference to a storyline fragment while "c0" and "i3" are labels identifying that section and structure embedded within it.

The symbol "kfx_id" is used for two different purposes. "kfx_id::" is an annotation that identifies what follows as a label. "kfx_id:" is the key that identifies the label that applies to a "structure" data structure.

Originally Posted by xxyzz

- The meaning of the numbers in section_position_id_map, for example, the "c117-spm" row:

$ion_1_0
section_position_id_map::{
  section_name: kfx_id::"c117",
  contains: [
    [
      1,
      kfx_id::"i11A"
    ],
    [
      2,
      kfx_id::"i11E"
    ],
    [
      3,
      kfx_id::"i11C"
    ],

I'm not sure what's 1, 2, 3.

Those are offsets used to calculate "position" numbers. Add them to the starting position number of that section to produce the actual position number that corresponds to the book content referenced by the associated kfx_id.

There are variations in how the section_position_id_map can be organized. See process_spim() within yj_position_location.py.

[xxyzz]

Thanks! Are there any other table, row or field not used in KFX file?

[jhowell]

Originally Posted by xxyzz

Thanks! Are there any other table, row or field not used in KFX file?

Fragments that are unreferenced will be excluded during conversion to KFX. Those are sometimes left when content is created and then deleted in Kindle Create. The gc_fragment_properties and gc_reachable tables may be related to this. I do not use them and so have not examined them in detail.

Nothing else comes to mind.