Kindle Scribe notebook format

[j.p.s]

Originally Posted by jhowell

I been tweaking my rendering of strokes as SVG and it is still a work in progress. I am attaching two images. The first is the PNG from your Scribe showing Amazon's rendering of your sample page. The second is a PNG produced by rasterizing the SVG produced by my plugin. There are noticeable differences especially in the highlighter and pencil strokes.

I think the highlights are better in your implementation. Some of the amazon fountain pen renders look a little better than yours, the others it is hard to say one way or other.

I doubt that it possible to exactly match pencil. Yours looks fine to me.

[tomsem]

Yes, it's amazing you've gotten it to look this close to the original. And the gain is that it's something you can archive and edit with any number of ePub editing tools.

[jhowell]

I released a new version of the KFX Input plugin for calibre that can convert Scribe notebooks into EPUB files containing SVG images. Instructions are in the first post of the KFX Input thread.

There is certainly room for improvement. Let me know of any suggestions or bug reports.

This is a one-way process. I have no intention of producing software that can create or modify Scribe notebooks to incorporate externally sourced content.

[tomsem]

Originally Posted by jhowell

I released a new version of the KFX Input plugin for calibre that can convert Scribe notebooks into EPUB files containing SVG images. Instructions are in the first post of the KFX Input thread.

There is certainly room for improvement. Let me know of any suggestions or bug reports.

This is a one-way process. I have no intention of producing software that can create or modify Scribe notebooks to incorporate externally sourced content.

Thanks, will check it out!

[jhowell]

Here is some documentation of my reverse engineering of the implementation of Scribe notebooks.

Scribe notebooks are in KDF format which contains KFX content in an SQLite database with embedded "fingerprint" data to deter reading or modification by non-Amazon software.

Hand drawn content is contained within a canvas (page). The information for each page is:

• Canvas width (pixels)
• Canvas height (pixels)
• PPI (pixels per inch)

Resolution is always 2520 ppi (99.2 pixels per mm). Notebook pages are 15624 by 20832 pixels (6.2 by 8.27 inches). Sticky notes are 13726 by 7350 pixels (5.45 by 2.92 inches).

Each stroke is composed of:

• Color - always zero (black) so far
• Brush type - 0=original pen, 1=highlighter, 5=pencil, 6=fountain pen, 7=new pen (if constant thickness) or marker (if variable thickness)
• Random seed - for generating random pencil density
• Stroke bounds - X/Y coordinate limits of the stroke in pixels (upper left and lower right), measured from top left of the canvas
• Thickness - in pixels, chosen from a set of five values that are different for each brush type
• List of points - see below

It has not yet been confirmed whether points are generated at a constant rate while a stroke is being created. The highest measured rate is approximately 368 points per second, however that might include interpolated points added for smoothness. Subsequent points at the same position are eliminated from the list in newer firmware.

The information provided for each point is:

• X position - offset from left side of stroke bounds
• Y position - offset from top of stroke bounds
• Thickness adjustment factor - multiplier for stroke thickness (percent)
• Density adjustment factor - amount of random fill for pencil strokes (percent)
• X tilt - angle in degrees, -63 (left) to +63 (right), 0 is upright
• Y tilt - angle in degrees, -63 (up) to +63 (down), 0 is upright
• Pressure - 1 (light) to 4095 (heavy)

The first four of these can be used to replicate the strokes as rendered by the Scribe fairly well. The other three values (tilt and pressure) are not needed directly for rendering but are instead raw data from the digitizer that go into the calculation of the two adjustment factors.


Derivation of stroke thickness and density adjustment factors from raw data

The Scribe user selects one of five stroke thicknesses (fine, thin, medium, thick, or heavy). The base thickness in pixels for each selection varies by brush type. The actual thickness of drawn strokes is also influenced by other factors and can vary from point to point along the stroke. The density of the stroke, the amount of dithered fill, can likewise vary.

The handing of stroke thickness and density depends on the brush type.

• Stroke thickness and density do not vary for pen and highlighter.
• Stroke thickness for marker and pencil takes into account the pressure applied and the angle that the pen is tilted from upright.
• Stroke density for pencil also uses the same factors but they influence it differently. Density is constant for all other brush types.
• Stroke thickness for fountain pen uses the above factors and also the direction of motion of the pen along the screen.

The raw data of pressure, tilt, and direction of motion from the pen are each used to derive a factor. The transformation of raw data into each factor is performed by computing a cubic Bézier curve, allowing non-linear behavior. Slightly different curves are used for each brush type and selected base thickness.

All of the factors that apply to a particular brush type are multiplied by the base thickness selected by the user to calculate the actual thickness produced along the stroke. Different factors are computed for stroke density and these are also combined by multiplication.

While the Bézier curves used to transform raw data are all different there is some commonality and how they function.

• Higher pressure produces thicker strokes. Changes in pressure at mid-level pressures have little effect. The effect of changes in pressure on line thickness is greatest at both low and high pressure.
  
• More tilt from upright (0 degrees) toward flat against the screen (90 degrees) produces thicker but less dense strokes. The effect tilt increases as the tilt angle increases. The direction that the pen is tilted (left, right, up, down) has no effect, only the angle with respect to the screen is significant.
  
• Direction of motion produces thinner strokes when moving toward the top of the screen and thicker strokes when moving toward the bottom of the screen. Moving left or right is neutral.

Examples of the Bézier curves are attached.

[jhowell]

There are still a few things I would like to discover about Scribe notebooks.

A sample notebook with strokes produced using carefully measured time durations would help to determine the rate at which points are digitized.

Examples of strokes with the pen held at different known orientations (tilt angles) would help to clarify how those values are stored. Which directions are positive and negative?

It would be helpful if someone would explain how the fountain pen works. Is the stroke thickness controlled by pen angle, pressure, speed of drawing, or a combination of these? Is pen angle relative to stroke direction a factor?

[tomsem]

In terms of sampling rates, I'm going to try video recording a stroke, then we can get a more accurate number. And also tilt angle samples (ideally after finding a protractor).

Fountain Pen is not pressure sensitive. Marker and Pencil are.

Fountain pen is like a thin line segment at about a fixed 20 degree angle. The thinness varies over a narrow range with the stroke 'weight'. The segment length is determined by one of the five 'weights'. So minimum thickness when direction aligns with 20 degrees, maximum thickness is achieved when direction is aligned with 110 degrees. (Seems this is biased to right handers, there should be one with minus 20 degree pitch as well! Will this discrimination never end?!)

Pencil is only type with tilt.

Attached samples:

• notebook with Pencil strokes with various weights, pressures, and tilt, and Fountain pen strokes at different angles (the 'circles' are at least 2 strokes, but hopefully capture all angles for each 'weight')
• notebook with single stroke, video recorded
• the video recording of stroke
• PDF export for each notebook

The stroke started at about 2 37/60 seconds (60fps) and ended 5 41/60 s so 3 4/60 seconds total.

[jhowell]

Originally Posted by tomsem

Fountain Pen is not pressure sensitive. Marker and Pencil are.

Fountain pen is like a thin line segment at about a fixed 20 degree angle. The thinness varies over a narrow range with the stroke 'weight'. The segment length is determined by one of the five 'weights'. So minimum thickness when direction aligns with 20 degrees, maximum thickness is achieved when direction is aligned with 110 degrees. (Seems this is biased to right handers, there should be one with minus 20 degree pitch as well! Will this discrimination never end?!)

Pencil is only type with tilt.

Thanks for the info. I am currently trying to correlate raw digitizer data (tilt and pressure) with the stroke thickness and density that appear to be generated based on them. That knowledge will be helpful in forming hypotheses.

Originally Posted by tomsem

Attached samples:

• notebook with Pencil strokes with various weights, pressures, and tilt, and Fountain pen strokes at different angles (the 'circles' are at least 2 strokes, but hopefully capture all angles for each 'weight')
• notebook with single stroke, video recorded
• the video recording of stroke
• PDF export for each notebook

Thanks for doing all of that work!

It will take me some time to analyze the notebook with all of the various samples. Already I have found several things that break my previous assumptions. Many of those strokes have density and thickness far beyond anything I saw previously.

Also that notebook contains two templates, vertical_rule_margin_1860x2480 and dotted_1860x2480, with only the second being active. I did not expect that and my KFX Input plugin currently does not handle it correctly. Did you possibly start with one template and then switch to another in that notebook?

Originally Posted by tomsem

The stroke started at about 2 37/60 seconds (60fps) and ended 5 41/60 s so 3 4/60 seconds total.

The stroke in the notebook matching the video contains 885 points. With a duration of 3.07 seconds that means the digitizer captures at least 288 points per second.

In the samples I have analyzed I have never found two subsequent points at the same location so it appears that those are eliminated to save space. That means the actual number of points originally generated by the digitizer was likely greater than found in the notebook and the actual sampling rate is likely higher. I am guessing that it is 300 samples per second but it could be even more.

If you are interested in pursuing this further then a different test could be done to get a more accurate measurement. For example a stroke could be made with the pen moving more quickly, already in motion before pressing down and lifting it, so that points are not eliminated near the ends. Another idea would be to first draw two lines and then quickly draw a new stroke that crosses them. The time at which the new stroke crosses the others could be detected on video and also within the captured data. That would mitigate against the potential loss of points near the ends of the measured stroke.

Those are just some ideas. I understand if you believe that you have already put enough effort into this.

[tomsem]

Yes, I changed the template mid-stream. Interesting that it keeps a copy of both.

I'm up for doing another video.

[jhowell]

Originally Posted by tomsem

Attached samples:

• notebook with Pencil strokes with various weights, pressures, and tilt, and Fountain pen strokes at different angles (the 'circles' are at least 2 strokes, but hopefully capture all angles for each 'weight')

Thanks again for all of the samples. I am starting with the first page (pencil strokes). Let me know if any of the following assumptions are incorrect.

I assume that L/M/H refers to low, medium, and high pressure. In the data the pressure was often maxed out (4095) even in the M stroke.

In the data the values for tilt varied in the X axis with straight up being 0 degrees and increasing in the positive direction. I need to know whether the top of the pen was increasingly tilted toward the left side or the right side of the screen to get the various thicknesses? (I am guessing to the right as would be more natural for a right-handed person.)

Thanks

[tomsem]

Originally Posted by jhowell

Thanks again for all of the samples. I am starting with the first page (pencil strokes). Let me know if any of the following assumptions are incorrect.

I assume that L/M/H refers to low, medium, and high pressure. In the data the pressure was often maxed out (4095) even in the M stroke.

In the data the values for tilt varied in the X axis with straight up being 0 degrees and increasing in the positive direction. I need to know whether the top of the pen was increasingly tilted toward the left side or the right side of the screen to get the various thicknesses? (I am guessing to the right as would be more natural for a right-handed person.)

Thanks

Yes, pressure is approximate, and hard to reproduce from set to set. Tilt is to the right side.

Here's another sampling test (2 line version). Focus is a little blurry crossing the first line (hand got in the way...) but hope it is good enough.

[jhowell]

Originally Posted by tomsem

Here's another sampling test (2 line version). Focus is a little blurry crossing the first line (hand got in the way...) but hope it is good enough.

Here is my analysis.

VLC shows a frame rate of 59.96 fps. I counted 43 frames between the crossing of the two lines for a duration of 0.72 sec.

Using Inkscape on the generated SVG I found the range of interest in the Y direction to be from 6370 to 15955. I then filtered the points by that range and verified that the line was truncated properly at either end. That yielded a total of 265 points.

Combining these values yields a report rate of 368 points per second.

This is quite a bit higher than I expected. I may have messed up somewhere.

[tomsem]

Let's call it an even 360 pps; a nice Babylonian number.

[tomsem]

Is it possible to decode the files containing text notes, highlights, bookmarks (in .sdr folder)? I assume it is likely SQLite DB but using db client it opens, but get 'there are no entities'. Also tried KFX Input as .kfx and .kpf

Also, I finally got around to comparing raw .ion with the .svg for one of the templates. I was surprised there appear to be no 'primitives' used for the .ion entities, such that they could be mapped to an SVG graphics element like line, text, ellipse, rect etc.

I had been assuming they'll be adding more object types (Shape, Text etc), and that the templates would show some examples, and retaining object identity, etc. But no, it's turtles all the way down.

I guess I should have noticed this in the Share PDF, where the apparent template text isn't selectable as text.

Doesn't mean they won't add this feature in the future but it makes me somewhat pessimistic that it will arrive in the next couple of months.

[j.p.s]

Originally Posted by tomsem

Is it possible to decode the files containing text notes, highlights, bookmarks (in .sdr folder)?

See KRDS, yet another jhowell masterpiece

https://www.mobileread.com/forums/sh...d.php?t=322172