Kindle Scribe notebook format

[jhowell]

Originally Posted by tomsem

This is the PDF export for the Notebook of post #23.

As you will see, it just contains raster images: no objects or layers.

Thanks for that.

Currently I can closely replicate the top two lines. My version of the middle line comes out too thin and I don't understand why yet. I need to work on replicating the speckling of the fourth (pencil) line. And for the bottom (highlighter) line the ends look different on my replication but theirs doesn't look very nice either. I will continue tweaking my code.

Originally Posted by tomsem

To the extent we can convert the on-device notebooks (to PDF or FXL ePub), it would be an improvement, since we would be able to preserve more of the original properties, and presumably be able to have template and annotation on separate layers (something Amazon should be doing IMO, and maybe they will, some day).

I am going to produce separate SVG files for the drawing and the template and set the template SVG as the background. That will make it easy to separate them.

Originally Posted by tomsem

I'll get some Pen annotation data for documents/books next (with export PDF).

I will look at those next.

[tomsem]

I do not know if this is at all helpful, but this is SVG exported from Illustrator of a single pencil like stroke.

[jhowell]

Originally Posted by tomsem

ebooks: .sdr and .documents folders (content is copy-protected)
kfx_document: public domain book .sdr, .documents folder, .kfx
print_replica_document: documents folder, .sdr, .kfx

By examining your samples I was able to determine how sticky note and full page annotations are handled.

As previously determined each book with those types of annotation has an associated notebook. For page annotations the notebook pages are full screen sized. For sticky notes the notebook pages are smaller. Each notebook page (aka section in KFX terminology) is identified by a unique section id string, such as "cJ6mymITFQPGtCVEx3QRvnA0".

The .yjr file in the .sdr folder associated with the book includes annotation information. Each annotation contains a reference to a start and end position in the book where the annotation applies. For regular annotations there is associated text or a highlight color. For Scribe annotations there is instead a reference to the section id of the notebook page containing the annotation image.

In some of the samples you provided the .sdr folder also has a file with the extension ".yjr.bad_file" containing a truncated version of the .yjr file. This is possibly the result of a bug in the Scribe firmware.

Originally Posted by tomsem

I do not know if this is at all helpful, but this is SVG exported from Illustrator of a single pencil like stroke.

I believe that my handling of the pencil brush type is good enough for now. If I revisit it I will take a look at that SVG for some ideas on how to do it better.

In about a week I plan to release a version of the KFX Input plugin that can convert Scribe notebooks to EPUBs containing SVG images. I just have a few more things to wrap up.

[tomsem]

Originally Posted by jhowell

By examining your samples I was able to determine how sticky note and full page annotations are handled.

As previously determined each book with those types of annotation has an associated notebook. For page annotations the notebook pages are full screen sized. For sticky notes the notebook pages are smaller. Each notebook page (aka section in KFX terminology) is identified by a unique section id string, such as "cJ6mymITFQPGtCVEx3QRvnA0".

The .yjr file in the .sdr folder associated with the book includes annotation information. Each annotation contains a reference to a start and end position in the book where the annotation applies. For regular annotations there is associated text or a highlight color. For Scribe annotations there is instead a reference to the section id of the notebook page containing the annotation image.

In some of the samples you provided the .sdr folder also has a file with the extension ".yjr.bad_file" containing a truncated version of the .yjr file. This is possibly the result of a bug in the Scribe firmware.



I believe that my handling of the pencil brush type is good enough for now. If I revisit it I will take a look at that SVG for some ideas on how to do it better.

In about a week I plan to release a version of the KFX Input plugin that can convert Scribe notebooks to EPUBs containing SVG images. I just have a few more things to wrap up.

Nice work!

A couple of questions:

- is the Notebook title stored in notebook and decodable? (I'm seeing kindle_title_metadata in there but the properties are encoded and straight uudecode didn't work for me)

- have you worked out how to render templates?

Looking forward to the update.

[jhowell]

Originally Posted by tomsem

Nice work!

Thanks. And thank you for providing all of the data files.

Originally Posted by tomsem

- is the Notebook title stored in notebook and decodable? (I'm seeing kindle_title_metadata in there but the properties are encoded and straight uudecode didn't work for me)

The only thing in the kindle_title_metadata for Scribe notebooks is the book_id, which is a random string (a base64 equivalent to a UUID) used as part of the calculation of the "fingerprint" that is designed to prevent those files from being read or altered by non-Amazon software.

Actually in many of the notebooks you provided the fingerprint appears to be set incorrectly. It is correct in some of the notebooks which leads me to believe that I am checking it properly. This is possibly another Scribe bug.

Originally Posted by tomsem

- have you worked out how to render templates?

Templates can already be decoded. You can find them in the Scribe firmware in the /app/standalonenotebook/templates folder. The templates in the current 5.16.1.3 firmware are:

checklist_1_col_1860x2480
daily_planner_1860x2480
dotted_1860x2480
graph_1860x2480
habit_tracker_vertical_1860x2480
medium_rule_1860x2480
medium_rule_margin_1860x2480
monthly_planner_1860x2480
music_1860x2480
narrow_rule_1860x2480
narrow_rule_margin_1860x2480
storyboard_1860x2480
vertical_rule_1860x2480
vertical_rule_margin_1860x2480
weekly_planner_1860x2480
wide_rule_1860x2480
wide_rule_margin_1860x2480

Each is a folder containing a single book.kdf file. If you zip one of those folders and then change the file extension from .zip to .kfx-zip you can decode it using the command line interface of the current version of the KFX Input plugin.

C:\Test>calibre-debug -r "KFX Input" -- narrow_rule_margin_1860x2480.kfx-zip
Software versions: KFX Input 1.51.0, calibre 6.17, Windows-10-10.0.22621-SP0
KFX Input plugin help is available at https://www.mobileread.com/forums/showthread.php?t=291290

Processing narrow_rule_margin_1860x2480.kfx-zip
Processing container: narrow_rule_margin_1860x2480.kfx-zip
Removed 1 KDF SQLite file fingerprint(s)
Adding book_navigation
Built approximate location_map with 1 locations
Features: CanonicalFormat-1, max_id-825, reflow-style-1
Metadata: asset_id=CR!32EMPXC3MFCK6THUW8RQ3Y4WKGHJ, book_id=Eil4x9xgS8ed9MPRIfijhQ0, creator_version=2.15.0, file_creator=YJConversionTools, is_sample=False, kfxgen=kfxlib-20230206, override_kindle_font=False, reading_orders=1, selection=enabled
Converting narrow_rule_margin_1860x2480.kfx-zip to EPUB
Converting book to EPUB 3
Converted book saved to narrow_rule_margin_1860x2480.epub

The resulting EPUB will have a single XHTML file containing the SVG image of the page template. (The EPUB does not render well in calibre but if you save the XHTML file and open it in a web browser you can see the template.)

[luketheduke]

Originally Posted by jhowell

Templates can already be decoded. You can find them in the Scribe firmware in the /app/standalonenotebook/templates folder.

Ooooo, looks like the Scribe's notebook application is make with the new React Native KPP framework.

Could you tell me the contents of `/app/standalonenotebook/js/`?

[j.p.s]

Originally Posted by luketheduke

Ooooo, looks like the Scribe's notebook application is make with the new React Native KPP framework.

Could you tell me the contents of `/app/standalonenotebook/js/`?

Presumably you can download the firmware yourself and extract it with kindletool.

[luketheduke]

Originally Posted by j.p.s

Presumably you can download the firmware yourself and extract it with kindletool.

Yeah, I'm just lazy and don't want to deal with ext4 filesystems rn.

I'm curious what would happen when trying to run the Scribe's software on a device like the PW3, though, so I'll probably just do it anyway.

I wonder if older devices will ever get v5.16.x...

[luketheduke]

Originally Posted by luketheduke

Could you tell me the contents of `/app/standalonenotebook/js/`?

Nevermind, doesn't exist.

[PoP]

Originally Posted by jhowell

...
The resulting EPUB will have a single XHTML file containing the SVG image of the page template. (The EPUB does not render well in calibre but if you save the XHTML file and open it in a web browser you can see the template.)

...And for example, if you convert (from update_kindle_scribe_5.16.2 firmware) the daily_planner_1860x2480 resulting epub to AZW3 (in Calibre) it renders properly on the PW3

[tomsem]

Originally Posted by jhowell

Thanks. And thank you for providing all of the data files.



The only thing in the kindle_title_metadata for Scribe notebooks is the book_id, which is a random string (a base64 equivalent to a UUID) used as part of the calculation of the "fingerprint" that is designed to prevent those files from being read or altered by non-Amazon software.

So is there any way get the Title of the notebook?

[PoP]

Originally Posted by luketheduke

... I'm curious what would happen when trying to run the Scribe's software on a device like the PW3 ...

indesScribable

[jhowell]

Originally Posted by tomsem

So is there any way get the Title of the notebook?

Not that I have seen. It is probably stored somewhere else along with the notebook folder information.

[tomsem]

As you may have seen, they have updated Print Replica support so annotations can include highlights and text notes. I don't think it changes much with respect to this project, but can upload a sample if it would be helpful.

[tomsem]

Originally Posted by jhowell

Not that I have seen. It is probably stored somewhere else along with the notebook folder information.

I will do an experiment to confirm this: back up file, change title, restore file. If title does not revert, then we can say it's not stored in the notebook in some sneaky way.