Kindle Scribe notebook format

[jhowell]

Originally Posted by Bluebotlabs

The actual NMDL data itself: ... Seems to be base64 encoded binary... unfortunately since it's encoded binary even after decoding it's not exactly easy to understand...

You are correct, that is base64. That is how binary blobs are presented in Ion text format. In the actual notebook file that data is binary.

I was able to work out what that data represents by trial and error testing. It is a compressed list of integer values. The ad hoc compression scheme used takes advantage of the fact that sequential points on a stroke are very close in value to each other.

The decoder for that data will be part of the upcoming KFX Input calibre plugin release which will be able to decode Scribe notebooks.

Originally Posted by Bluebotlabs

Additionally, something I found interesting, KDF files did exist before the scribe... but there's very little information on them

https://www.kdpcommunity.com/s/quest...language=en_US

Yes, KDF files are the main component of KPF (Kindle Package Format) produced by Amazon's Kindle Create software. The main difference between KDF and KFX is that KFX is a compact binary format and must be completely rewritten if it needs to be modified. KDF on the other hand is stored in an SQLite database and can be updated on-the-fly.

Originally Posted by Bluebotlabs

Perhaps it is ION?
https://amazon-ion.github.io/ion-docs/

Ion is the coding that underlies both KFX and KDF formats. It has a compact binary representation that is used in those formats. It also has an equivalent textual representation (somewhat like JSON) that can be used when Ion content needs to be understood. The code shown in the spoiler in this post is KDF data presented in Ion text format.

[GeorgeYellow]

Originally Posted by jhowell

I did notice one change in the latest Scribe firmware, handling of "orphan" notebooks. That appears to refer to notebook files found in the .notebooks folder that are not expected to be on the device. As far as I can tell the firmware will now delete orphan notebooks when they are detected.

Speaking of the firmware, have you looked into what 'Datamate' is?
There's a `.sync/datamateConfig.json' which seems to allow changing the sync URL (`datamateUrl`) as well as using Curl to upload (`datamateUseCurl`).

Any chance of local synchronization?

[HackerDude]

Originally Posted by jhowell

You are correct, that is base64. That is how binary blobs are presented in Ion text format. In the actual notebook file that data is binary.

I was able to work out what that data represents by trial and error testing. It is a compressed list of integer values. The ad hoc compression scheme used takes advantage of the fact that sequential points on a stroke are very close in value to each other.

The decoder for that data will be part of the upcoming KFX Input calibre plugin release which will be able to decode Scribe notebooks.



Yes, KDF files are the main component of KPF (Kindle Package Format) produced by Amazon's Kindle Create software. The main difference between KDF and KFX is that KFX is a compact binary format and must be completely rewritten if it needs to be modified. KDF on the other hand is stored in an SQLite database and can be updated on-the-fly.



Ion is the coding that underlies both KFX and KDF formats. It has a compact binary representation that is used in those formats. It also has an equivalent textual representation (somewhat like JSON) that can be used when Ion content needs to be understood. The code shown in the spoiler in this post is KDF data presented in Ion text format.

Ah, that makes a lot of sense, thanks!

Do you think you could convert all the templates to PDFs? I need them for a project but I can't do it myself :\

[jhowell]

Originally Posted by GeorgeYellow

Speaking of the firmware, have you looked into what 'Datamate' is?
There's a `.sync/datamateConfig.json' which seems to allow changing the sync URL (`datamateUrl`) as well as using Curl to upload (`datamateUseCurl`).

Any chance of local synchronization?

From examining strings in the Scribe firmware it appears that "Datamate" refers to an Amazon server involved in notebook synchronization. The default URL seems to be "https://sync.datamate.kindle.amazon.dev/graphql". That is all I know about it.

Perhaps you could change the URL in the JSON file to point to your own server but unless Amazon goofed badly you will not be able to establish a connection due to SSL security. Even if you can connect, working out the details of the synchronization protocol is going to be difficult. And if Amazon finds out you are doing it then that file will likely be moved somewhere inaccessible in a future firmware release.

[jhowell]

Originally Posted by Bluebotlabs

Do you think you could convert all the templates to PDFs? I need them for a project but I can't do it myself :\

If you have a Scribe then you should be able to create several notebooks with a blank page, each based on a different template, and export them as PDF.

If you don't have a Scribe then some sample images from templates are available in the file attached to this post by tomsem. See the thumbnails folder.

If you need all of the templates then I suggest the following procedure:

• Use kindletool to extract the disk image from the most recent Scribe firmware.
• Locate the templates in the "/app/standalonenotebook/templates" folder.
• Rename each "book.kdf" file to "templatename.kfx".
• Use the CLI of the KFX Input plugin for calibre to convert each of those to an EPUB.
• Print the EPUB to a PDF file using calibre.

Good luck!

[HackerDude]

Originally Posted by jhowell

If you have a Scribe then you should be able to create several notebooks with a blank page, each based on a different template, and export them as PDF.

If you don't have a Scribe then some sample images from templates are available in the file attached to this post by tomsem. See the thumbnails folder.

If you need all of the templates then I suggest the following procedure:

• Use kindletool to extract the disk image from the most recent Scribe firmware.
• Locate the templates in the "/app/standalonenotebook/templates" folder.
• Rename each "book.kdf" file to "templatename.kfx".
• Use the CLI of the KFX Input plugin for calibre to convert each of those to an EPUB.
• Print the EPUB to a PDF file using calibre.

Good luck!

Oh lol, I didn't realise I could just rename the kdf files to kfx (at least for the templates) and then just use Calibre with your plugin

Thanks!

[HackerDude]

Originally Posted by jhowell

If you have a Scribe then you should be able to create several notebooks with a blank page, each based on a different template, and export them as PDF.

If you don't have a Scribe then some sample images from templates are available in the file attached to this post by tomsem. See the thumbnails folder.

If you need all of the templates then I suggest the following procedure:

• Use kindletool to extract the disk image from the most recent Scribe firmware.
• Locate the templates in the "/app/standalonenotebook/templates" folder.
• Rename each "book.kdf" file to "templatename.kfx".
• Use the CLI of the KFX Input plugin for calibre to convert each of those to an EPUB.
• Print the EPUB to a PDF file using calibre.

Good luck!

One more thing, do you think it would be possible to get SVGs from them? or are they raster based...

[jhowell]

Originally Posted by Bluebotlabs

One more thing, do you think it would be possible to get SVGs from them? or are they raster based...

Templates are vector based. The EPUB file created from a template by the KFX Input plugin will have a single HTML page containing an SVG image.

[HackerDude]

Originally Posted by jhowell

Templates are vector based. The EPUB file created from a template by the KFX Input plugin will have a single HTML page containing an SVG image.

Totally forgot EPUB as HTML based lol
This is perfect for my needs, thanks!

[jhowell]

I am still working on the release of the KFX Input plugin with support for Scribe notebooks.

One thing that would help to complete it is a notebook with an example of each possible thickness for each brush type. Something line one page per brush type with a stroke of each thickness drawn on it.

Another thing I saw in the latest Scribe firmware release was a mention of three stroke colors: light, dark, and black. I don't know how these are selected in the user interface but it would be good to have examples of those also.

[PoP]

Originally Posted by jhowell

Another thing I saw in the latest Scribe firmware release was a mention of three stroke colors: light, dark, and black. I don't know how these are selected in the user interface but it would be good to have examples of those also.

Perhaps from the pressure sensitive pen, not the interface?

[tomsem]

Originally Posted by jhowell

I am still working on the release of the KFX Input plugin with support for Scribe notebooks.

One thing that would help to complete it is a notebook with an example of each possible thickness for each brush type. Something line one page per brush type with a stroke of each thickness drawn on it.

Another thing I saw in the latest Scribe firmware release was a mention of three stroke colors: light, dark, and black. I don't know how these are selected in the user interface but it would be good to have examples of those also.

Here you go.

I do not see any 'stroke colors' for any of the line types or for the highlighter. It can't be the pressure measure either, since that has a larger range of values.

There is at least a need for other colors (including white) when writing on black/dark ('PDF') backgrounds. But I don't see it anywhere. Maybe they'll turn this on one day when we're not paying attention. (I assume Dark Mode does not change any properties and just automatically inverts color)

[jhowell]

Originally Posted by tomsem

Here you go.

Thanks for that sample. It helped to fill in some of the gaps in my understanding.

I been tweaking my rendering of strokes as SVG and it is still a work in progress. I am attaching two images. The first is the PNG from your Scribe showing Amazon's rendering of your sample page. The second is a PNG produced by rasterizing the SVG produced by my plugin. There are noticeable differences especially in the highlighter and pencil strokes.

Originally Posted by tomsem

I do not see any 'stroke colors' for any of the line types or for the highlighter. It can't be the pressure measure either, since that has a larger range of values.

There is at least a need for other colors (including white) when writing on black/dark ('PDF') backgrounds. But I don't see it anywhere. Maybe they'll turn this on one day when we're not paying attention. (I assume Dark Mode does not change any properties and just automatically inverts color)

It definitely was a menu choice in the firmware. It is likely something that isn't finished yet or that they decided to not release for some reason.

[tomsem]

Originally Posted by jhowell

It definitely was a menu choice in the firmware. It is likely something that isn't finished yet or that they decided to not release for some reason.

I suppose it is supposed to be some palette hanging off the toolbar (under the stroke thickness perhaps?).

[HackerDude]

Originally Posted by jhowell

Thanks for that sample. It helped to fill in some of the gaps in my understanding.

I been tweaking my rendering of strokes as SVG and it is still a work in progress. I am attaching two images. The first is the PNG from your Scribe showing Amazon's rendering of your sample page. The second is a PNG produced by rasterizing the SVG produced by my plugin. There are noticeable differences especially in the highlighter and pencil strokes.



It definitely was a menu choice in the firmware. It is likely something that isn't finished yet or that they decided to not release for some reason.

From what I can tell, yes, definitely a toolbar of sorts, I mean, there's literally a folder called `toolbar`

I think it is also important to note that, especially apparent for the highlighter, the angle and direction of the pen can also influence the stroke, I don't know if you've already implemented this or if its just in the data but just saying
Another thing you may want to look into is the beta Scribe web-viewer which can display notes in the web page, however, I suspect it merely uses the PDF-export feature...

https://read.amazon.com/kindle-notebook


Another thing which could be worthwhile could be looking at whatever handles PDF conversion, if it's on the kindle itself...