KRDS - A parser for Kindle reader data store files

[j.p.s]

Originally Posted by willemml

I have tested editing annotations and it works.

As far as I know you are the first to announce an ability to write to these files.

[willemml]

I have now split the repo into two libraries, one provides the file definitions while the other provides the methods for serializing and deserializing. There is documentation in the file definition repo for the fields that are "known". If I missed anything let me know (either here or make an issue, or even better a PR.)

File definitions:
https://github.com/willemml/kindle_formats-rs

Serializer and deserializer (read and write to KRDS files):
https://github.com/willemml/serde_krds

(This is just a small update on what I am doing. There is no functionality added, although now these libraries are published on crates.io (ser/de, definitions) and serde_krds conforms to the Serde data format crate conventions.

[paulequilibrio]

Originally Posted by willemml

I have now split the repo into two libraries, one provides the file definitions while the other provides the methods for serializing and deserializing. There is documentation in the file definition repo for the fields that are "known". If I missed anything let me know (either here or make an issue, or even better a PR.)

File definitions:
https://github.com/willemml/kindle_formats-rs

Serializer and deserializer (read and write to KRDS files):
https://github.com/willemml/serde_krds

(This is just a small update on what I am doing. There is no functionality added, although now these libraries are published on crates.io (ser/de, definitions) and serde_krds conforms to the Serde data format crate conventions.

First of all, thank you so much for doing this!

I'm not a Rust developer, I've already tried cargo build but I believe it's not creating an executable that I can run. Could you add some instructions on how to use your code to the REAMED file in the repository?

[willemml]

Originally Posted by paulequilibrio

First of all, thank you so much for doing this!

I'm not a Rust developer, I've already tried cargo build but I believe it's not creating an executable that I can run. Could you add some instructions on how to use your code to the REAMED file in the repository?

These are libraries, not executables, I created these so it would be easier to create binary programs in the future. There are tests that can be run (using cargo test) but other than that this is purely a library.

[j.p.s]

I would like to convert the line endings for krds.py on github from \r\n to \n

Are there any objections?

Rationale is as follows:

When I invoke krds.py directly under linux instead of invoking python with krds.py as an argument, it dies with:

krds.py: /usr/bin/python3^M: bad interpreter: No such file or directory

which is a complaint that the file with a carriage return as the end if its name does not exist.

I searched around and mostly found blogs automatically generated documentation sites that discussed reading and writing cross platform text files and only mentioned python script line endings in passing.

They all agreed that python itself tries to be agnostic about line endings, but that best practice is to use the UNIX style LF for line endings.

I don't know what windows does when (trying) to execute a python script directly, but I'm guessing that it ignores the #! line then crashes. I hope that macs work with the \n ending and maybe they treat the \r as white space, but maybe the script crashes for the same reason as linux.

In any case, the only use for the shebang (#!) line is to allow direct script execution. So the line as is should either be removed, its ending changed to \n only, or the entire script changed to LF only line endings. For the above reasons I would like to do the last. (Besides, it save 398 bytes.)

I am not a python person and don't use microsoft or apple products, so I might be overlooking something. If so, let me know.

ETA:
I tried limiting my search to python.org and found
https://discuss.python.org/t/page-au...g-solved/26644
so I'm also inclined to change the shebang line to

#! /usr/bin/env python3

[tomsem]

I've updated krds.py to handle the new underline (so far Scribe-only) annotation type:

    ANNOT_CLASS_NAMES = {
        0: "annotation.personal.bookmark",
        1: "annotation.personal.highlight",
        2: "annotation.personal.note",
        3: "annotation.personal.clip_article",  # value not verified
        10: "annotation.personal.handwritten_note",
        11: "annotation.personal.sticky_note",
        13: "annotation.personal.underline",
    }

It appears Active Canvas notes use type 10 ('annotation.personal.handwritten_note'), just as they do in Print Replica.

JSON output example for each:

  "annotation.cache.object": {
       [ ... ]
       "annotation.personal.underline": [
            {
                "startPosition": "AcsPAAB6AQAA:342909",
                "endPosition": "AcsPAACaAQAA:342941",
                "creationTime": "2025-01-20T15:16:39.771000",
                "lastModificationTime": "2025-01-20T15:16:39.771000",
                "template": "0\ufffc0"
            }
        ],
        "annotation.personal.handwritten_note": [
            {
                "startPosition": "AcsPAAAAAAAA:342531",
                "endPosition": "AcsPAAAAAAAA:342531",
                "creationTime": "2025-01-20T15:20:36.451000",
                "lastModificationTime": "2025-01-20T15:20:36.451000",
                "template": "0\ufffc0",
                "handwritten_note_nbk_ref": "clWfVWeqbQviRtOrSGYnf0A0"
            }

I think there was a 3rd highlight type (Outline? Box? - like what Kindle 1 did?) that was referenced in recent update files but has not been enabled. Guessing that would be type '12' if and when it shows up.

[j.p.s]

Originally Posted by tomsem

I've updated krds.py to handle the new underline (so far Scribe-only) annotation type

I've pushed this addition to the KRDS repository at github, which has bug fixes and other improvements.
https://github.com/K-R-D-S/KRDS

It was a bit tedious since for some reason you weren't working from that. So I'm also attaching it here in the hope that you will base any future changes on it.

[tomsem]

Originally Posted by j.p.s

I've pushed this addition to the KRDS repository at github, which has bug fixes and other improvements.
https://github.com/K-R-D-S/KRDS

It was a bit tedious since for some reason you weren't working from that. So I'm also attaching it here in the hope that you will base any future changes on it.

Did not know there was a repo. I will use it next time they change things.

Scribe is supposed to get something called 'expandable book margins' in 'early 2025' but I don't think it will add any annotation types.

Thanks.

[j.p.s]

Originally Posted by tomsem

Did not know there was a repo. I will use it next time they change things.

Scribe is supposed to get something called 'expandable book margins' in 'early 2025' but I don't think it will add any annotation types.

Thanks.

Oh. I didn't know that you didn't know.

I've been kind of hoping that jhowell and you join it as contributor or even owner, since I'm not comfortable with being the sole person with write access, especially since I haven't been much of a contributor.

[tomsem]

Originally Posted by j.p.s

Oh. I didn't know that you didn't know.

I've been kind of hoping that jhowell and you join it as contributor or even owner, since I'm not comfortable with being the sole person with write access, especially since I haven't been much of a contributor.

I've Starred and Watched the repo.

[jhowell]

Originally Posted by j.p.s

I've been kind of hoping that jhowell and you join it as contributor or even owner, since I'm not comfortable with being the sole person with write access, especially since I haven't been much of a contributor.

I received the request to become an owner. Thanks.

[aakar]

I've been trying to figure out how to view synced highlights for my personal documents. Especially because I like to use multiple devices to read throughout the day. Thanks to @jhowell for his script and the KFX Input Plugin and some finagling with ChatGPT CodeX, I feel like I've landed on something that works.

This is only valuable for those who are emailing personal documents to their Kindle and highlighting on multiple devices. You can send the HTML file to Readwise, which was my entire reason for doing this.

I hadn't found anyone trying to do this, so I can only assume it's a small subset of people.

Here's the repo. I'm sure there are a ton of bugs, seeing as how AI helped to write it, but it's worked for me for a few personal documents.