KRDS - A parser for Kindle reader data store files

[gesundheit]

Originally Posted by jhowell

KFX uses two values separated by a colon. The first is a base64 encoding of the eid and offset, which are fields used internally by KFX to determine the location of content.

What's the exact format for the decoded eid and offset bytestring?

I'm finding that, for example, b64 decoding the kfx position ATwIAABnAAAA:177266 yields 0x013c08000067000000. It seems that the first 4 bytes, 0x013c0800, are the eid, and the last 4 bytes, 0x67000000, are the offset, is that right?

But there is an byte between them: 0x013c08000067000000
What is this extra byte for?

Also, do you happen to know what template is?

        "annotation.personal.bookmark": [
            {
                "startPosition": "AVoDAAAAAAAA:9430",
                "endPosition": "AVoDAAAAAAAA:9430",
                "creationTime": "2019-08-11T15:24:03.088000",
                "lastModificationTime": "2019-08-11T15:24:03.088000",
                "template": "0\ufffc0"
            },

[jhowell]

Originally Posted by gesundheit

What's the exact format for the decoded eid and offset bytestring?

The first byte is always 0x01, probably there to allow for future variations. The next four bytes are the eid and the last four bytes are the offset.

The eid labels a section of KFX content. It is the KFX equivalent to an HTML id attribute. The offset counts positions from the start of that eid. The position following the colon is an overall position within the book and is another way to refer to the same content.

Positions in KFX count Unicode characters of visible text with each image also counted as one position.

Originally Posted by gesundheit

Also, do you happen to know what template is?

I have no idea what that is for. It appears to be the same in all cases I have seen. Unicode FFFC is the Object Replacement Character.

[gesundheit]

I see, I got it now. Thanks for the explanation.

[tomsem]

Looks like this needs to be updated for Scribe annotation type:

annotation.personal.sticky_note = 11

Are there really 4-10? Or those are reserved types?

Will check Print Replica next and supply data files, if not updated code.

Also got 'excess values' exception ('val==[0]', i.e. not empty as expected, probably because of this extra annotation?).

In meantime, I've patched to handle exceptions so I can look at what's working.

[j.p.s]

Maybe something like a github repository is in order.

[tomsem]

I was able to fix it up to handle the two new annotation types (subject to code review).

Main changes:

    ANNOT_CLASS_NAMES = {
        0: "annotation.personal.bookmark",
        1: "annotation.personal.highlight",
        2: "annotation.personal.note",
        3: "annotation.personal.clip_article",  # value not verified
        10: "annotation.personal.handwritten_note",
        11: "annotation.personal.sticky_note",
    }

Added 10 & 11.
(made this a class attribute to consolidate duplicated strings)

        elif name in self.ANNOT_CLASS_NAMES.values():
            obj["startPosition"] = self.decode_position(val.pop(0))
            obj["endPosition"] = self.decode_position(val.pop(0))
            obj["creationTime"] = datetime.datetime.fromtimestamp(val.pop(0) / 1000.0).isoformat()
            obj["lastModificationTime"] = datetime.datetime.fromtimestamp(val.pop(0) / 1000.0).isoformat()
            obj["template"] = val.pop(0)

            if name == "annotation.personal.note":
                obj["note"] = val.pop(0)
            elif name == "annotation.personal.handwritten_note":
                obj["handwritten_note_nbk_ref"] = val.pop(0)
            elif name == "annotation.personal.sticky_note":
                obj["sticky_note_nbk_ref"] = val.pop(0)

Added handwritten_note_nbk_ref property. I verified that these match to a symbol in ion_symbol_table in the nbk with the pen annotations for a page of the Print Replica document.

I also could not keep myself from fixing linter complaints about 'unreferenced variable i' (by replacing with '_').

Scribe also seems to have another piece of data in font_prefs structure (value 0). I added 'unknown2' to absorb it. I checked another Kindle and it does not have this.

    "font.prefs": {
        "typeface": "_INVALID_,und:bookerly",
        "lineSp": 1,
        "size": 3,
        "align": 0,
        "insetTop": 64,
        "insetLeft": 200,
        "insetBottom": 8,
        "insetRight": 200,
        "unknown1": -1,
        "bold": 1,
        "userSideloadableFont": "",
        "customFontIndex": -1,
        "mobi7SystemFont": "",
        "mobi7RestoreFont": false,
        "readingPresetSelected": "",
        "unknown2": 0
    },

Maybe is page animation flag? I'll do some more checking.

Not sure what I'm going to do with this, but excited to have discovered it. Thanks @jhowell & @j.p.s.!

[jhowell]

Originally Posted by tomsem

I was able to fix it up to handle the two new annotation types (subject to code review).

I have changed the first post in this thread to eliminate the original krds.py and link to your updated version.

Originally Posted by j.p.s

Maybe something like a github repository is in order.

Not worth the bother for me to do it. If someone else wants to create that I will change the first post of this thread to link to it.

[j.p.s]

Originally Posted by j.p.s

Maybe something like a github repository is in order.

Originally Posted by jhowell

Not worth the bother for me to do it. If someone else wants to create that I will change the first post of this thread to link to it.

I'm willing to create the repository and commit the original version (with attribution) and tomsem's changes (with attribution). I would probably split the latter into 3 commits:

1. s/i/_/
2. changes needed to handle the new annotation types
3. formatting changes (mostly white space)

I would also want to add j-howell and tomsem's github account (if he has one) as contributors. I assume that it would be tomsem making all or most of any future commits.

[jhowell]

Originally Posted by j.p.s

I'm willing to create the repository and commit the original version (with attribution) and tomsem's changes (with attribution). I would probably split the latter into 3 commits:

1. s/i/_/
2. changes needed to handle the new annotation types
3. formatting changes (mostly white space)

I would also want to add j-howell and tomsem's github account (if he has one) as contributors. I assume that it would be tomsem making all or most of any future commits.

That is fine with me.

[j.p.s]

Originally Posted by j.p.s

I'm willing to create the repository.

I'm inclined to create a new organization for this. It turns out KRDS (unrelated) is already taken.

How about KRDS-parse, KRDS-utils or some such? Suggestions welcome.

The repository itself can be KRDS.

ETA: K-R-D-S is available, which is my new preferred org name.

[j.p.s]

I ended up having to restore the original white space to tomsem's version in order to see what the actual coding changes were. I ran jhowell's original version, tomsem's version, and my final version against the yj* files tomsem uploaded for The Experience Machine and the sodoku books. The JSON files exactly matched for tomsem's and my versions.

All versions generated the non fatal error:
"ERROR: Unknown data structure whisperstore.migration.status"
for both yjf files.

[j.p.s]

The KRDS repository is up at:
https://github.com/K-R-D-S/KRDS
with releases available as tar.gz and zip at:
https://github.com/K-R-D-S/KRDS/tags

[jhowell]

Originally Posted by j.p.s

The KRDS repository is up at:
https://github.com/K-R-D-S/KRDS
with releases available as tar.gz and zip at:
https://github.com/K-R-D-S/KRDS/tags

Thanks!

[j.p.s]

You're welcome.

I hope it will be useful to anyone developing KRDS further.

[willemml]

I decided to learn how to use the popular Rust library serde.rs by making a full deserializer and serializer library for the KRDS format (this is sort of a practice run before I attempt a full KPF and KFX serializer/deserializer.) You just have to define the format you are trying to read as a Rust structure. I have already done this for yjr, yjf, azw3r and azw3f files. It should be very easy to add support for any other KRDS files that come up as all that needs to be done is define field names.

The code for this is published in a GitHub repo. https://github.com/willemml/krds_serde

There is a discrepancy in yjr and azw3r files in that the Kindle (in my case the Scribe) serializes empty strings as "0x03 0x00 0x00 x00" whereas yjf (and azw3f) files serialize them as "0x03 0x01" I have chosen to use this shorter format for all files (and the Kindle does not complain, I have tested editing annotations and it works.) Other than that binary output of the serializer is identical to an input binary. The code is currently rough and undocumented, but it works and has tests.