Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Readers > Amazon Kindle > Kindle Developer's Corner

Notices

Reply
 
Thread Tools Search this Thread
Old 05-13-2023, 05:38 PM   #1
tomsem
Grand Sorcerer
tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.
 
Posts: 6,163
Karma: 22803835
Join Date: Apr 2009
Location: USA
Device: iPhone 14PM, Kindle Scribe, iPad mini 6, PocketBook Era
Kindle Scribe notebook format

I'm curious about how Scribe stores Notebook data, synching behavior, etc, and plan to engage in some experimentation.

It appears that everything is stored in SQLite databases in KPF containers, each with a -journal to manage atomic commit and rollback (undo/redo/sync to cloud?). There is also a thumbnails folder that has a .png render of the first page of each downloaded notebook.

Folder structure is like this snapshot of my Scribe:

Quote:
.notebooks
- guid-like named sub-folders e.g. "d088f501-48f6-7a6b-341c-726964927fff"
(1 for each downloaded notebook)
- thumbnails
(PNG thumbnail for each downloaded notebook, uses same filename as the notebook folder they correspond to)
- .backup (mine is empty)
You might see folders with '!!EBOK!!' or '!!PDOC!!' in the name. These appear to contain Pen annotations for store content and personal documents, respectively, the content itself would be in /documents folder.

@jhowell:
Quote:
The templates for Scribe notebooks are contained in the firmware and stored in individual subfolders under /app/standalonenotebook/templates. These are in KDF format which is a variant of KFX that uses an SQLite database.
@jhowell on folders:
Quote:
It is likely that they are handled the same way as collections, as entries in an SQLite database stored in a part of the file system that is inaccessible via USB.
The nbk data can be exported to NMDL ('Note Model Description Language', expressed in Amazon ion data format) using KFX input plugin.

Findings:
* it is possible to backup and restore specific notebook versions
* it is probably possible to export to SVG once NMDL is sufficiently 'understood'

Last edited by tomsem; 05-15-2023 at 12:37 PM.
tomsem is offline   Reply With Quote
Old 05-13-2023, 05:49 PM   #2
GeorgeYellow
Enthusiast
GeorgeYellow will become famous soon enoughGeorgeYellow will become famous soon enoughGeorgeYellow will become famous soon enoughGeorgeYellow will become famous soon enoughGeorgeYellow will become famous soon enoughGeorgeYellow will become famous soon enough
 
Posts: 30
Karma: 500
Join Date: Nov 2017
Device: Nook, Kindle
To contrast

Kindle Scribe - no network connection since one-time-registration.

Creating a local notebook.

Root: .notebooks

there's an empty .backups directory
there's a thumbnails directory with the PNG

Then, there's a GUID directory per notebook.
Since its not connected to the network, 'nbk' is populated, and nbk-journal is 0-bytes.


* Defining Folders doesn't result in a change to the user-store.

* I haven't tweaked anything in KFX_CONFIG yet
GeorgeYellow is offline   Reply With Quote
Advert
Old 05-13-2023, 08:30 PM   #3
tomsem
Grand Sorcerer
tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.
 
Posts: 6,163
Karma: 22803835
Join Date: Apr 2009
Location: USA
Device: iPhone 14PM, Kindle Scribe, iPad mini 6, PocketBook Era
Looks like it is more like: a GUID folder per downloaded Notebook, though that was not reflected in the earlier snapshot.

Still not clear where folders are defined. It's not one of the GUID folders as those are all accounted for as containers for Notebook content, and after adding another I don't see any changes.

I'll try to keep initial post up to date.

Last edited by tomsem; 05-13-2023 at 09:26 PM.
tomsem is offline   Reply With Quote
Old 05-14-2023, 07:55 AM   #4
jhowell
Grand Sorcerer
jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.
 
jhowell's Avatar
 
Posts: 6,251
Karma: 82988365
Join Date: Nov 2011
Location: Tampa Bay, Florida
Device: Oasis 2, iPad, Nexus 7
Quote:
Originally Posted by tomsem View Post
There are 18 templates. I thought they might be represented by the !!EBOK!! folders somehow but not clear how since they are not 1 to 1. And it would make more sense not to have them in user storage area.
The page templates for notebooks are are stored in subfolders under /app/standalonenotebook/templates, which is not accessible via USB. They are in KDF format, which is a variant of KFX stored in an SQLite database.

Quote:
Originally Posted by tomsem View Post
The other mystery is where folders are defined. I do not notice any new files when adding new folders.
It is likely that they are handled the same way as collections, as entries in an SQLite database stored in a part of the file system that is inaccessible via USB.

Quote:
Originally Posted by tomsem View Post
So far I have not been able to open any of the data files with my SQL browser (Beekeeper Studio for Mac), or rather it says 'there are no entities'. Maybe they are compressed and need to be expanded.
Amazon's KDF format uses an SQLite database but with additional fingerprint data that prevents access or modification without using the proper software. It is possible that these database use the same mechanism.

If someone would post an example of one of these files I could check.

Last edited by jhowell; 05-14-2023 at 08:29 AM.
jhowell is offline   Reply With Quote
Old 05-14-2023, 10:42 AM   #5
jhowell
Grand Sorcerer
jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.
 
jhowell's Avatar
 
Posts: 6,251
Karma: 82988365
Join Date: Nov 2011
Location: Tampa Bay, Florida
Device: Oasis 2, iPad, Nexus 7
Some information is known about Scribe annotation storage from previous examination.

Books and personal documents in KFX format contain a .yjr file found in the .sdr folder associated with the book. For non-Scribe devices that contains all of the annotations made to that book. On a Scribe the .yjr file also has links to handwritten annotations. Those are stored as strokes in an SQLite database which can be found in a folder associated with the book:
/.notebooks/{ASIN}!!{CDETYPE}!!notebook/nbk

Amazon may have used the same mechanism for annotating notebooks. If so then it is possible that each notebook has main KFX file holding a copy of the template for each page and an associated .sdr folder with a .yjr file linking to the SQLite stroke database for that notebook.

It might be worthwhile to search for such a folder structure, possible in a hidden subfolder of the documents folder of the Scribe. Or perhaps these are stored somewhere else, such as in the GUID named folders under .notebooks.
jhowell is offline   Reply With Quote
Advert
Old 05-14-2023, 10:58 AM   #6
jhowell
Grand Sorcerer
jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.
 
jhowell's Avatar
 
Posts: 6,251
Karma: 82988365
Join Date: Nov 2011
Location: Tampa Bay, Florida
Device: Oasis 2, iPad, Nexus 7
The templates for Scribe notebooks are contained in the firmware and stored in individual subfolders under /app/standalonenotebook/templates. These are in KDF format which is a variant of KFX that uses an SQLite database.

I extracted a template named daily_planner_1860x2480 from the Scribe firmware. I was able to convert this template to an HTML equivalent using the CLI mode of my KFX Input plugin.

The template contains a single page with an SVG image. This makes sense as it allows scaling without pixelation.

The zipped KDF and HTML files are attached.
Attached Files
File Type: zip daily_planner_1860x2480.zip (84.8 KB, 51 views)

Last edited by jhowell; 05-14-2023 at 11:01 AM.
jhowell is offline   Reply With Quote
Old 05-14-2023, 11:07 AM   #7
tomsem
Grand Sorcerer
tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.
 
Posts: 6,163
Karma: 22803835
Join Date: Apr 2009
Location: USA
Device: iPhone 14PM, Kindle Scribe, iPad mini 6, PocketBook Era
So it looks like the !!EBOK!! and !!PDOC!! folders are actually Pen annotations for ebooks and personal documents, respectively.

Last edited by tomsem; 05-14-2023 at 11:24 AM.
tomsem is offline   Reply With Quote
Old 05-14-2023, 11:09 AM   #8
tomsem
Grand Sorcerer
tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.
 
Posts: 6,163
Karma: 22803835
Join Date: Apr 2009
Location: USA
Device: iPhone 14PM, Kindle Scribe, iPad mini 6, PocketBook Era
Here is a zipped up folder for a notebook.
Attached Files
File Type: zip 61233b77-330b-5427-4581-5f977d998bb5.zip (62.4 KB, 88 views)

Last edited by tomsem; 05-14-2023 at 11:19 AM.
tomsem is offline   Reply With Quote
Old 05-14-2023, 11:35 AM   #9
tomsem
Grand Sorcerer
tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.
 
Posts: 6,163
Karma: 22803835
Join Date: Apr 2009
Location: USA
Device: iPhone 14PM, Kindle Scribe, iPad mini 6, PocketBook Era
Quote:
Originally Posted by jhowell View Post
Some information is known about Scribe annotation storage from previous examination.

Books and personal documents in KFX format contain a .yjr file found in the .sdr folder associated with the book. For non-Scribe devices that contains all of the annotations made to that book. On a Scribe the .yjr file also has links to handwritten annotations. Those are stored as strokes in an SQLite database which can be found in a folder associated with the book:
/.notebooks/{ASIN}!!{CDETYPE}!!notebook/nbk

Amazon may have used the same mechanism for annotating notebooks. If so then it is possible that each notebook has main KFX file holding a copy of the template for each page and an associated .sdr folder with a .yjr file linking to the SQLite stroke database for that notebook.

It might be worthwhile to search for such a folder structure, possible in a hidden subfolder of the documents folder of the Scribe. Or perhaps these are stored somewhere else, such as in the GUID named folders under .notebooks.
Notebooks are 100% Pen annotations, applied to a particular template, right? The template doesn't change, so there's no need to have a copy of it, just a reference to the one used for the notebook. I looked for hidden folders/files in /documents, and there aren't any.
tomsem is offline   Reply With Quote
Old 05-14-2023, 11:53 AM   #10
j.p.s
Wizard
j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.
 
Posts: 4,945
Karma: 91769388
Join Date: Apr 2011
Device: pb360
Quote:
Originally Posted by tomsem View Post
Notebooks are 100% Pen annotations, applied to a particular template, right? The template doesn't change, so there's no need to have a copy of it, just a reference to the one used for the notebook. I looked for hidden folders/files in /documents, and there aren't any.
So the templates are perfect right off the bat and will not evolve over time?
j.p.s is offline   Reply With Quote
Old 05-14-2023, 11:59 AM   #11
tomsem
Grand Sorcerer
tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.
 
Posts: 6,163
Karma: 22803835
Join Date: Apr 2009
Location: USA
Device: iPhone 14PM, Kindle Scribe, iPad mini 6, PocketBook Era
It appears you cannot back up and restore older notebook versions using computer.

This worked:
- copy notebook folder to computer
- remove notebook folder from Scribe
- disconnect usb and sync (download is gone)
- connect usb and copy notebook folder back
- disconnect usb
- download is restored

This didn't work:
- copy notebook folder
- disconnect usb
- make a change to notebook and sync
- remove download
- connect usb
- copy previous notebook folder to scribe
- disconnect usb
- no download shown, it ignores older version

So to backup a version, it looks like you need to make a copy that also gets saved to cloud.

Last edited by tomsem; 05-14-2023 at 12:11 PM.
tomsem is offline   Reply With Quote
Old 05-14-2023, 12:08 PM   #12
tomsem
Grand Sorcerer
tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.tomsem ought to be getting tired of karma fortunes by now.
 
Posts: 6,163
Karma: 22803835
Join Date: Apr 2009
Location: USA
Device: iPhone 14PM, Kindle Scribe, iPad mini 6, PocketBook Era
Quote:
Originally Posted by j.p.s View Post
So the templates are perfect right off the bat and will not evolve over time?
That would be my guess.

If they actually wanted to deprecate or replace a template they could hide the old one so it cannot be used to create new notebooks but would still be there for notebooks that referenced the deprecated template.

But not clear they'll ever need to update them. They probably will add new ones over time (e.g. 'wide' aspect templates).
tomsem is offline   Reply With Quote
Old 05-14-2023, 03:07 PM   #13
jhowell
Grand Sorcerer
jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.
 
jhowell's Avatar
 
Posts: 6,251
Karma: 82988365
Join Date: Nov 2011
Location: Tampa Bay, Florida
Device: Oasis 2, iPad, Nexus 7
Quote:
Originally Posted by tomsem View Post
Here is a zipped up folder for a notebook.
That is definitely a KDF file, KFX format in an SQLite database with a "fingerprint" to prevent tampering. I tried to convert it using my KFX Input plugin but it uses many new data types that I have not encountered before so very little of it can be decoded at present. It appears to be three pages of content based on the music_1860x2480 template.

I have attached a textual representation of the data contained in that file in Amazon's Ion data format.
Attached Files
File Type: txt nbk.txt (322.7 KB, 43 views)

Last edited by jhowell; 05-14-2023 at 03:11 PM.
jhowell is offline   Reply With Quote
Old 05-14-2023, 03:23 PM   #14
jhowell
Grand Sorcerer
jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.
 
jhowell's Avatar
 
Posts: 6,251
Karma: 82988365
Join Date: Nov 2011
Location: Tampa Bay, Florida
Device: Oasis 2, iPad, Nexus 7
Quote:
Originally Posted by j.p.s View Post
So the templates are perfect right off the bat and will not evolve over time?
The sample notebook file provided by tomsem references the music_1860x2480 template but it also includes a copy of that template internally. So it appears that once a notebook is created it will not be affected by any future changes made to the template it is based on.
jhowell is offline   Reply With Quote
Old 05-14-2023, 03:32 PM   #15
j.p.s
Wizard
j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.
 
Posts: 4,945
Karma: 91769388
Join Date: Apr 2011
Device: pb360
Quote:
Originally Posted by jhowell View Post
The sample notebook file provided by tomsem references the music_1860x2480 template but it also includes a copy of that template internally. So it appears that once a notebook is created it will not be affected by any future changes made to the template it is based on.
I misunderstood post #9 to say that a notebook does not include a copy of the template.
j.p.s is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Kindle Scribe jhowell Amazon Kindle 681 09-11-2023 07:41 PM
Kindle scribe-Stylus HLS Amazon Kindle 6 05-03-2023 10:00 AM
Kindle Scribe Sonnenfee Amazon Kindle 0 01-04-2023 08:25 AM
Kindle Scribe - DS chrispe Amazon Kindle 11 12-26-2022 11:10 PM
Amazon announces the Kindle Scribe - 10.2" Kindle with stylus support Dazrin News 128 12-22-2022 02:16 PM


All times are GMT -4. The time now is 12:06 AM.


MobileRead.com is a privately owned, operated and funded community.