[Conversion Plugin] KFX Input

[jhowell]

Originally Posted by tomsem

repeats previous location

I have spent several weeks working on and off to understand how the Scribe derives the thickness and density adjustment factors from the more primitive data (x/y coordinates, tilt angle, and pressure applied.) Earlier this week I had a breakthrough and was able to exactly replicate the thickness and density of the hundreds of thousands of points in the Scribe notebook samples I have seen so far.

I did some more testing and that one notebook sample you provided (95E4A4423B184AE1B3DCF747F659B0CD!!PDOC!!notebook) breaks all of those rules. Do you have any idea what might be different about that notebook as compared with all of the other notebooks you have shared so far?

[tomsem]

Originally Posted by jhowell

I have spent several weeks working on and off to understand how the Scribe derives the thickness and density adjustment factors from the more primitive data (x/y coordinates, tilt angle, and pressure applied.) Earlier this week I had a breakthrough and was able to exactly replicate the thickness and density of the hundreds of thousands of points in the Scribe notebook samples I have seen so far.

I did some more testing and that one notebook sample you provided (95E4A4423B184AE1B3DCF747F659B0CD!!PDOC!!notebook) breaks all of those rules. Do you have any idea what might be different about that notebook as compared with all of the other notebooks you have shared so far?

I do have an idea.

The document in this case started life as a PDF containing 8 Sudoku puzzles. I generally 'permanently delete' these as soon as I have solved the puzzles. This removes it from cloud, as well as the KFX, .sdr folder, and (in this case) the associated .notebook folder.

The fact that this folder had been orphaned suggests that it was deleted on my other Scribe. This was probably while I was doing experiments to see how sync of pen annotations worked in various scenarios.

So this document may have been one that I annotated with both Scribes. It may be some situation due to sync sequencing. I remember one case (but maybe not this one) where I had two solutions overlaid, for example.

Or it might be a simpler matter of having done a Delete Page operation along the way, and having things in undo buffer when document was closed?

At any rate, these are some scenarios that might factor in here (I'll try the Delete Page one to see what results)

Note that Permanently Delete does not remove personal documents from all devices, just from the one that initiated it, and from cloud storage. So you need to also Delete from the other device(s). It still seems like this should not orphan anything however.

(A few weeks ago I wrote a Python script to clean up orphaned .sdr folders, empty folders, and content specific temp files. At the time I had not observed any of these in .notebook folder so it does not look for any. Clearly I need to update it now...)

[PoP]

Originally Posted by jhowell

...several weeks...hundreds of thousands of points...

Kudos for a truly remarquable reverse engineering!

[jhowell]

Originally Posted by PoP

Kudos for a truly remarquable reverse engineering!

Thanks. However I wasted a lot more time than I should have trying to find a simple formula that fit the data. In the end I went though the actual Scribe firmware to discover that it uses cubic Bézier curves to map raw data from the pen into the factors that control the thickness and density of strokes.

[jhowell]

Originally Posted by jhowell

That notebook uses a brush type (value 0 internally) that I have not encountered before. Strokes for that are missing some of the data that is present for other brush types causing the plugin to fail.

After further research I found that this is caused by using the "pen" brush type in the original Scribe firmware. It was changed in later firmware. The next plugin update will handle this properly.

Originally Posted by jhowell

I did some more testing and that one notebook sample you provided (95E4A4423B184AE1B3DCF747F659B0CD!!PDOC!!notebook) breaks all of those rules.

This was not as bad as I first thought. After further testing I was able to determine that the strokes in that notebook have a slight jitter applied to the x/y coordinates of points that was throwing off my calculations. I am assuming that this Sudoku puzzle was annotated using a previous Scribe firmware version with that feature. The next plugin update will handle this properly.

-----

At this point I believe that I have accounted for all of the problems in the last set of sample notebooks and will release the updated plugin after a bit of further testing. Thanks again for all of the help.

[jhowell]

Version 2.2.0 - 10 Jul 2023

Miscellaneous improvements in the conversion of Scribe notebooks including handling of empty notebooks, unreferenced pages, original firmware "pen" brush type, repeated point locations, and local_delta_fragments.

[exy]

Conversion fails with following error for one book. is there any solution?
Latest Calibre and plugins on Win.


WARNING: This book contains PDF content, which can be extracted using the KFX Input plugin CLI.
ERROR: This book has a layout that is incompatible with calibre conversion. For best results use the KFX Input plugin CLI for conversion.
Converting book to EPUB 3
Format is fixed layout
Traceback (most recent call last):
File "calibre_plugins.kfx_input.__init__", line 115, in convert
Exception: This book has a layout that is incompatible with calibre conversion. For best results use the KFX Input plugin CLI for conversion.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "runpy.py", line 196, in _run_module_as_main
File "runpy.py", line 86, in _run_code
File "site.py", line 83, in <module>
File "site.py", line 78, in main
File "site.py", line 50, in run_entry_point
File "calibre\utils\ipc\worker.py", line 215, in main
File "calibre\gui2\convert\gui_conversion.py", line 38, in gui_convert_override
File "calibre\gui2\convert\gui_conversion.py", line 25, in gui_convert
File "calibre\ebooks\conversion\plumber.py", line 1108, in run
File "calibre\customize\conversion.py", line 242, in __call__
File "calibre_plugins.kfx_input.__init__", line 126, in convert
calibre.ebooks.conversion.ConversionUserFeedBack: {"msg": "<b>Cannot convert ABC - X : Pqr stu vwxyz</b><br><br>Exception('This book has a layout that is incompatible with calibre conversion. For best results use the KFX Input plugin CLI for conversion.')", "level": "error", "det_msg": "", "title": "KFX conversion failed"}

[jhowell]

Originally Posted by exy

WARNING: This book contains PDF content, which can be extracted using the KFX Input plugin CLI.
ERROR: This book has a layout that is incompatible with calibre conversion. For best results use the KFX Input plugin CLI for conversion.

The calibre conversion system is designed to handle reflowable books. Fixed layout (page layout that cannot be changed) is not supported.

See the Command Line Interface section in the first post of this thread for instructions on using the plugin's CLI to convert fixed layout books.

[jhowell]

Version 2.3.0 - 07 Aug 2023

Handle new heading level metadata. (Fixes "Unexpected Ion symbols used: $798, $799, $800" and "nav_container xxx has unknown type: $798")

Fix error when processing the cover of fixed layout books converted from KPF format. (Fixes "YJFragmentList item is missing: '$389'")

[xvicarious]

This plugin seems to treat KPF and KFX files the same. Say if I have a KPF file, and try to use your other plugin to convert to to KFX, it will create an intermediate EPUB file, while the KFX Output plugin handles KPF files.

[jhowell]

Originally Posted by xvicarious

This plugin seems to treat KPF and KFX files the same. Say if I have a KPF file, and try to use your other plugin to convert to to KFX, it will create an intermediate EPUB file, while the KFX Output plugin handles KPF files.

The plugins have different functionality depending on whether they are used as part of a calibre conversion process or invoked using their command line interfaces. Calibre conversion always uses EPUB as the intermediate format regardless of the selected input and output formats. The plugin CLIs do more direct conversion.

KPF is closely related to KFX. Either can be used as an input format for the KFX Input plugin. And the CLI of KFX Output can transform KPF directly into KFX.

[willemml]

Hello, I am trying to write a script that will let me extract PDFs from the Kindle in their annotated form without contacting amazon. Do the annotation "notebooks" (each write-on PDF on the Kindle seems to have a corresponding notebook folder that when put through KFX Input gives me the SVG of all my annotations.) Do these "notebooks" contain info on which SVG goes with which page of each PDF? (Even if it is not in the corresponding epub file and only tells me which pen traces go on each page in the KFX nbk file.) If so where is this data stored?

[jhowell]

Originally Posted by willemml

Hello, I am trying to write a script that will let me extract PDFs from the Kindle in their annotated form without contacting amazon. Do the annotation "notebooks" (each write-on PDF on the Kindle seems to have a corresponding notebook folder that when put through KFX Input gives me the SVG of all my annotations.)

The notebook folder name is based on the metadata of the KFX book being annotated. It is composed of the content_id, cde_content_type, and the string "notebook"; all separated by "!!". For example "EBEA035E6DB444159EF42DA7E5EEF8F6!!PDOC!!notebook" .

Originally Posted by willemml

Do these "notebooks" contain info on which SVG goes with which page of each PDF? (Even if it is not in the corresponding epub file and only tells me which pen traces go on each page in the KFX nbk file.) If so where is this data stored?

The EPUB produced by KFX Input from a Scribe annotation notebook contains one XHTML file per annotation, each linking to an SVG image. The connection between a book page (in KFX format produced from PDF) and the associated annotation notebook page is provided by a file with the extension .yjr found in the .sdr folder associated with the KFX book. That file can be converted to JSON using KRDS - A parser for Kindle reader data store files.

Each annotated page will have an entry such as:

    "annotation.cache.object": {
        "annotation.personal.handwritten_note": [
            {
                "startPosition": "201.0:13974",
                "endPosition": "201.0:13974",
                "creationTime": "2023-08-26T09:09:38.130000",
                "lastModificationTime": "2023-08-26T09:09:38.130000",
                "template": "0\ufffc0",
                "handwritten_note_nbk_ref": "crEq-GhRTSa63nk5j3KC6Qw0"
            }
        ]
    },

The startPosition is a KFX position number that corresponds to the book page being annotated. The page number can be found by looking up the part of the position number following the colon in a content JSON file that can be optionally produced by the CLI of the KFX Input plugin. (The number will match a type 2 entry. Count type 2 entries in the file to find the page number.)

The handwritten_note_nbk_ref is the KFX section ID of the associated annotation page in the notebook. Currently those IDs are not reflected in the EPUB generated by the KFX Input plugin for an annotation notebook. I will update the plugin to include this data in the EPUB so that these can be matched.

The margins of the PDF page may be been trimmed during conversion to KFX format for delivery to the Scribe. Also the SVG produced will have the aspect ratio of the Scribe screen which might not match the PDF page. Because of this some image manipulation may be needed to properly overlay the SVG image onto the original PDF page.

[willemml]

Originally Posted by jhowell

The notebook folder name is based on the metadata of the KFX book being annotated. It is composed of the content_id, cde_content_type, and the string "notebook"; all separated by "!!". For example "EBEA035E6DB444159EF42DA7E5EEF8F6!!PDOC!!notebook" .

This much I had mostly figured out myself, but thank you for confirming and giving me the correct names for each part.

Originally Posted by jhowell

The EPUB produced by KFX Input from a Scribe annotation notebook contains one XHTML file per annotation, each linking to an SVG image. The connection between a book page (in KFX format produced from PDF) and the associated annotation notebook page is provided by a file with the extension .yjr found in the .sdr folder associated with the KFX book. That file can be converted to JSON using KRDS - A parser for Kindle reader data store files.

Each annotated page will have an entry such as:

    "annotation.cache.object": {
        "annotation.personal.handwritten_note": [
            {
                "startPosition": "201.0:13974",
                "endPosition": "201.0:13974",
                "creationTime": "2023-08-26T09:09:38.130000",
                "lastModificationTime": "2023-08-26T09:09:38.130000",
                "template": "0\ufffc0",
                "handwritten_note_nbk_ref": "crEq-GhRTSa63nk5j3KC6Qw0"
            }
        ]
    },

The startPosition is a KFX position number that corresponds to the book page being annotated. The page number can be found by looking up the part of the position number following the colon in a content JSON file that can be optionally produced by the CLI of the KFX Input plugin. (The number will match a type 2 entry. Count type 2 entries in the file to find the page number.)

Ah, wonderful, this is exactly what I was looking for. Thank you. Good to know that pages are type 2 entries, I had vaguely determined this already from the JSON output of this plugin, but was not sure. Do you know what all the content types are? I have so far only come across content type 2, but I guess that is because currently all the files I am examining are from print replica books created from PDF files.

Originally Posted by jhowell

The handwritten_note_nbk_ref is the KFX section ID of the associated annotation page in the notebook. Currently those IDs are not reflected in the EPUB generated by the KFX Input plugin for an annotation notebook. I will update the plugin to include this data in the EPUB so that these can be matched.

Thank you, that will help a lot with my scripting.

Originally Posted by jhowell

The margins of the PDF page may be been trimmed during conversion to KFX format for delivery to the Scribe. Also the SVG produced will have the aspect ratio of the Scribe screen which might not match the PDF page. Because of this some image manipulation may be needed to properly overlay the SVG image onto the original PDF page.

If I were to extract the PDF from the KFX files I am generating I assume that would save me from dealing with trimming the PDFs for correct alignment? And is there a point I can align the SVG to (say top left or right or similar) of the PDF with an offset to account for aspect ratio change?

In the mean time I am trying to write a program that can convert from PDFs to write-on-able KFX files without going through the Kindle Create software (which for now means I am trying to create my own KPF files from scratch that contain the metadata to correctly map PDF pages) so that I can put them through the KFX Output plugin (which I realize still relies on Kindle Previewer for conversion, but I eventually want to write my own KPF to KFX converter as well.) I will be publishing all code on GitHub as soon as I have something that works a little bit. Is there any documentation outside of this thread and the code of the KFX plugins you have (which is the closest I could find to documentation on the KFX and KPF formats other than a rough overview) that could help me?

[jhowell]

Originally Posted by willemml

Good to know that pages are type 2 entries, I had vaguely determined this already from the JSON output of this plugin, but was not sure. Do you know what all the content types are? I have so far only come across content type 2, but I guess that is because currently all the files I am examining are from print replica books created from PDF files.

Type 1 is text, type 2 is an image. For a print replica KFX there is a single image associated with each PDF page. There may also be type 1 entries if the source PDF had a text layer that was included in the KFX version of the book.

Originally Posted by willemml

If I were to extract the PDF from the KFX files I am generating I assume that would save me from dealing with trimming the PDFs for correct alignment?

As far as I can tell the included PDF is essentially the same as the original PDF, at least in terms of its page dimensions. The four margins for cropping are instead expressed as KFX metadata associated with each PDF page image resource.

That cropping data is currently not handled by the KFX Input plugin. Something will need to be done about that.

Originally Posted by willemml

And is there a point I can align the SVG to (say top left or right or similar) of the PDF with an offset to account for aspect ratio change?

Each XHTML page of the EPUB converted from the annotation notebook has an embedded SVG image that references the external SVG image containing the stroke data. If the aspect ratio of the annotation does not match the PDF then the style associated with the embedded SVG will have the height, width, top, and left properties needed to adjust it to match the associated PDF page aspect ratio.

Originally Posted by willemml

In the mean time I am trying to write a program that can convert from PDFs to write-on-able KFX files without going through the Kindle Create software (which for now means I am trying to create my own KPF files from scratch that contain the metadata to correctly map PDF pages) so that I can put them through the KFX Output plugin (which I realize still relies on Kindle Previewer for conversion, but I eventually want to write my own KPF to KFX converter as well.) I will be publishing all code on GitHub as soon as I have something that works a little bit.

I am curious to see what you come up with.

Originally Posted by willemml

Is there any documentation outside of this thread and the code of the KFX plugins you have (which is the closest I could find to documentation on the KFX and KPF formats other than a rough overview) that could help me?

I do not know of any existing documentation other than that provided by Amazon for the underlying Ion data format.