Kindle Scribe notebook format

[tomsem]

I'm curious about how Scribe stores Notebook data, synching behavior, etc, and plan to engage in some experimentation.

It appears that everything is stored in SQLite databases in KPF containers, each with a -journal to manage atomic commit and rollback (undo/redo/sync to cloud?). There is also a thumbnails folder that has a .png render of the first page of each downloaded notebook.

Folder structure is like this snapshot of my Scribe:

.notebooks
- guid-like named sub-folders e.g. "d088f501-48f6-7a6b-341c-726964927fff"
(1 for each downloaded notebook)
- thumbnails
(PNG thumbnail for each downloaded notebook, uses same filename as the notebook folder they correspond to)
- .backup (mine is empty)

You might see folders with '!!EBOK!!' or '!!PDOC!!' in the name. These appear to contain Pen annotations for store content and personal documents, respectively, the content itself would be in /documents folder.

@jhowell:

The templates for Scribe notebooks are contained in the firmware and stored in individual subfolders under /app/standalonenotebook/templates. These are in KDF format which is a variant of KFX that uses an SQLite database.

@jhowell on folders:

It is likely that they are handled the same way as collections, as entries in an SQLite database stored in a part of the file system that is inaccessible via USB.

The nbk data can be exported to NMDL ('Note Model Description Language', expressed in Amazon ion data format) using KFX input plugin.

Findings:
* it is possible to backup and restore specific notebook versions
* it is probably possible to export to SVG once NMDL is sufficiently 'understood'

[GeorgeYellow]

Kindle Scribe - no network connection since one-time-registration.

Creating a local notebook.

Root: .notebooks

there's an empty .backups directory
there's a thumbnails directory with the PNG

Then, there's a GUID directory per notebook.
Since its not connected to the network, 'nbk' is populated, and nbk-journal is 0-bytes.


* Defining Folders doesn't result in a change to the user-store.

* I haven't tweaked anything in KFX_CONFIG yet

[tomsem]

Looks like it is more like: a GUID folder per downloaded Notebook, though that was not reflected in the earlier snapshot.

Still not clear where folders are defined. It's not one of the GUID folders as those are all accounted for as containers for Notebook content, and after adding another I don't see any changes.

I'll try to keep initial post up to date.

[jhowell]

Originally Posted by tomsem

There are 18 templates. I thought they might be represented by the !!EBOK!! folders somehow but not clear how since they are not 1 to 1. And it would make more sense not to have them in user storage area.

The page templates for notebooks are are stored in subfolders under /app/standalonenotebook/templates, which is not accessible via USB. They are in KDF format, which is a variant of KFX stored in an SQLite database.

Originally Posted by tomsem

The other mystery is where folders are defined. I do not notice any new files when adding new folders.

It is likely that they are handled the same way as collections, as entries in an SQLite database stored in a part of the file system that is inaccessible via USB.

Originally Posted by tomsem

So far I have not been able to open any of the data files with my SQL browser (Beekeeper Studio for Mac), or rather it says 'there are no entities'. Maybe they are compressed and need to be expanded.

Amazon's KDF format uses an SQLite database but with additional fingerprint data that prevents access or modification without using the proper software. It is possible that these database use the same mechanism.

If someone would post an example of one of these files I could check.

[jhowell]

Some information is known about Scribe annotation storage from previous examination.

Books and personal documents in KFX format contain a .yjr file found in the .sdr folder associated with the book. For non-Scribe devices that contains all of the annotations made to that book. On a Scribe the .yjr file also has links to handwritten annotations. Those are stored as strokes in an SQLite database which can be found in a folder associated with the book:
/.notebooks/{ASIN}!!{CDETYPE}!!notebook/nbk

Amazon may have used the same mechanism for annotating notebooks. If so then it is possible that each notebook has main KFX file holding a copy of the template for each page and an associated .sdr folder with a .yjr file linking to the SQLite stroke database for that notebook.

It might be worthwhile to search for such a folder structure, possible in a hidden subfolder of the documents folder of the Scribe. Or perhaps these are stored somewhere else, such as in the GUID named folders under .notebooks.

[jhowell]

The templates for Scribe notebooks are contained in the firmware and stored in individual subfolders under /app/standalonenotebook/templates. These are in KDF format which is a variant of KFX that uses an SQLite database.

I extracted a template named daily_planner_1860x2480 from the Scribe firmware. I was able to convert this template to an HTML equivalent using the CLI mode of my KFX Input plugin.

The template contains a single page with an SVG image. This makes sense as it allows scaling without pixelation.

The zipped KDF and HTML files are attached.

[tomsem]

So it looks like the !!EBOK!! and !!PDOC!! folders are actually Pen annotations for ebooks and personal documents, respectively.

[tomsem]

Here is a zipped up folder for a notebook.

[tomsem]

Originally Posted by jhowell

Some information is known about Scribe annotation storage from previous examination.

Books and personal documents in KFX format contain a .yjr file found in the .sdr folder associated with the book. For non-Scribe devices that contains all of the annotations made to that book. On a Scribe the .yjr file also has links to handwritten annotations. Those are stored as strokes in an SQLite database which can be found in a folder associated with the book:
/.notebooks/{ASIN}!!{CDETYPE}!!notebook/nbk

Amazon may have used the same mechanism for annotating notebooks. If so then it is possible that each notebook has main KFX file holding a copy of the template for each page and an associated .sdr folder with a .yjr file linking to the SQLite stroke database for that notebook.

It might be worthwhile to search for such a folder structure, possible in a hidden subfolder of the documents folder of the Scribe. Or perhaps these are stored somewhere else, such as in the GUID named folders under .notebooks.

Notebooks are 100% Pen annotations, applied to a particular template, right? The template doesn't change, so there's no need to have a copy of it, just a reference to the one used for the notebook. I looked for hidden folders/files in /documents, and there aren't any.

[j.p.s]

Originally Posted by tomsem

Notebooks are 100% Pen annotations, applied to a particular template, right? The template doesn't change, so there's no need to have a copy of it, just a reference to the one used for the notebook. I looked for hidden folders/files in /documents, and there aren't any.

So the templates are perfect right off the bat and will not evolve over time?

[tomsem]

It appears you cannot back up and restore older notebook versions using computer.

This worked:
- copy notebook folder to computer
- remove notebook folder from Scribe
- disconnect usb and sync (download is gone)
- connect usb and copy notebook folder back
- disconnect usb
- download is restored

This didn't work:
- copy notebook folder
- disconnect usb
- make a change to notebook and sync
- remove download
- connect usb
- copy previous notebook folder to scribe
- disconnect usb
- no download shown, it ignores older version

So to backup a version, it looks like you need to make a copy that also gets saved to cloud.

[tomsem]

Originally Posted by j.p.s

So the templates are perfect right off the bat and will not evolve over time?

That would be my guess.

If they actually wanted to deprecate or replace a template they could hide the old one so it cannot be used to create new notebooks but would still be there for notebooks that referenced the deprecated template.

But not clear they'll ever need to update them. They probably will add new ones over time (e.g. 'wide' aspect templates).

[jhowell]

Originally Posted by tomsem

Here is a zipped up folder for a notebook.

That is definitely a KDF file, KFX format in an SQLite database with a "fingerprint" to prevent tampering. I tried to convert it using my KFX Input plugin but it uses many new data types that I have not encountered before so very little of it can be decoded at present. It appears to be three pages of content based on the music_1860x2480 template.

I have attached a textual representation of the data contained in that file in Amazon's Ion data format.

[jhowell]

Originally Posted by j.p.s

So the templates are perfect right off the bat and will not evolve over time?

The sample notebook file provided by tomsem references the music_1860x2480 template but it also includes a copy of that template internally. So it appears that once a notebook is created it will not be affected by any future changes made to the template it is based on.

[j.p.s]

Originally Posted by jhowell

The sample notebook file provided by tomsem references the music_1860x2480 template but it also includes a copy of that template internally. So it appears that once a notebook is created it will not be affected by any future changes made to the template it is based on.

I misunderstood post #9 to say that a notebook does not include a copy of the template.