Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Formats > Kindle Formats

Notices

Reply
 
Thread Tools Search this Thread
Old 08-12-2019, 05:59 PM   #1
jhowell
Wizard
jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.
 
jhowell's Avatar
 
Posts: 2,915
Karma: 28068759
Join Date: Nov 2011
Location: Florida
Device: Oasis 2, Fire, iPad Air 2, Nexus 7
KRDS - A parser for Kindle reader data store files

A recent discussion prompted me to look into how annotations are stored on Kindle devices running recent firmware versions.

Information related to each book being read is saved in a pair of sidecar files in the book's .sdr folder. These files contain serialized data objects used by the e-book reader application. The first file contains objects that change with every page turn such as the last page read and reading timing. The second file contains less frequently changed data such as personal annotations, font & dictionary choices, and synced reading position.

The file extensions used depend on the book format:
  • KF8 (.azw3) format: .azw3f and .azw3r
  • KFX format: .yjf and .yjr
  • MOBI (.azw) format: .mbs and .mbp1
  • PDF format: .pdt and .pds
  • Topaz (.azw1) format: .tas and .tal

The data format appears to be proprietary to Amazon and is similar to the Amazon Ion Binary Encoding used by KFX. It encodes the name of each object being serialized along with a list of property values. Values each have an associated data type, such as integer or string. Decoding objects requires knowledge of the data structure associated with each class.


KRDS (Kindle Reader Data Store)

I have written a Python script to parse these files. The main function accepts an input file name, parses it into a Python data structure, and outputs the result as a human readable JSON file.

I reverse engineered the data structures for several classes commonly used by the Kindle reader, but it is likely that I missed some things. Reports of any file that is not handled properly are welcome.


Usage
Spoiler:

Download and unzip the attachment to this post to obtain "krds.py". It should run under recent versions of Python 2 or 3.

Code:
usage: python krds.py [-h] pathname

Convert Kindle reader data store files to JSON

positional arguments:
  pathname    Pathname to be processed (.azw3f, .azw3r, .mbp1, .mbs, .yjf, .yjr)

optional arguments:
  -h, --help  show this help message and exit
Enclose the name of the file to be converted in double quotes if it contains spaces.
The output file will have the same name with ".json" appended.



Sample Output
Spoiler:

Decoded .yjr file:
Code:
{
    "font.prefs": {
        "typeface": "_INVALID_,und:bookerly",
        "lineSp": 1,
        "size": 5,
        "align": 1,
        "insetTop": -1,
        "insetLeft": -1,
        "insetBottom": -1,
        "insetRight": -1,
        "unknown1": -1,
        "bold": 1,
        "userSideloadableFont": "",
        "customFontIndex": -1,
        "mobi7SystemFont": "",
        "mobi7RestoreFont": false,
        "readingPresetSelected": ""
    },
    "sync_lpr": true,
    "annotation.cache.object": {
        "annotation.personal.highlight": [
            {
                "startPosition": "ATwDAAAAAAAA:3803",
                "endPosition": "ATwDAAADAQAA:4062",
                "creationTime": "2019-08-11T15:24:03.083000",
                "lastModificationTime": "2019-08-11T15:24:03.083000",
                "template": "0\ufffc0"
            },
            {
                "startPosition": "AS0DAAAAAAAA:1696",
                "endPosition": "AS0DAADoAAAA:1928",
                "creationTime": "2019-08-11T15:24:03.088000",
                "lastModificationTime": "2019-08-11T15:24:03.088000",
                "template": "0\ufffc0"
            },
            {
                "startPosition": "AWsDAAAAAAAA:12846",
                "endPosition": "AW0DAAB7AQAA:13491",
                "creationTime": "2019-08-11T15:24:03.088000",
                "lastModificationTime": "2019-08-11T15:24:03.088000",
                "template": "0\ufffc0"
            },
            {
                "startPosition": "ATUDAAAAAAAA:1975",
                "endPosition": "ATsDAAAtAgAA:3802",
                "creationTime": "2019-08-11T15:24:03.088000",
                "lastModificationTime": "2019-08-11T15:24:03.088000",
                "template": "0\ufffc0"
            },
            {
                "startPosition": "AUQDAAAAAAAA:5510",
                "endPosition": "AUgDAAADAQAA:6194",
                "creationTime": "2019-08-11T15:24:03.083000",
                "lastModificationTime": "2019-08-11T15:24:03.083000",
                "template": "0\ufffc0"
            },
            {
                "startPosition": "ASsDAAAAAAAA:1477",
                "endPosition": "ASsDAABOAAAA:1555",
                "creationTime": "2019-08-11T15:24:03.088000",
                "lastModificationTime": "2019-08-11T15:24:03.088000",
                "template": "0\ufffc0"
            },
            {
                "startPosition": "AW8DAAAAAAAA:13552",
                "endPosition": "ASIEAABwAAAA:42227",
                "creationTime": "2019-08-11T15:24:03.030000",
                "lastModificationTime": "2019-08-11T15:24:03.030000",
                "template": "0\ufffc0"
            },
            {
                "startPosition": "AWkDAAAAAAAA:12350",
                "endPosition": "AWkDAADvAAAA:12589",
                "creationTime": "2019-08-11T15:24:03.088000",
                "lastModificationTime": "2019-08-11T15:24:03.088000",
                "template": "0\ufffc0"
            },
            {
                "startPosition": "AT8DAAAAAAAA:4154",
                "endPosition": "AUADAAAxAQAA:4745",
                "creationTime": "2019-08-11T15:24:03.088000",
                "lastModificationTime": "2019-08-11T15:24:03.088000",
                "template": "0\ufffc0"
            }
        ],
        "annotation.personal.note": [
            {
                "startPosition": "AUADAAAxAQAA:4745",
                "endPosition": "AUADAAAxAQAA:4745",
                "creationTime": "2019-08-11T15:24:03.083000",
                "lastModificationTime": "2019-08-11T15:24:03.083000",
                "template": "0\ufffc0",
                "note": "Here is another note for the book"
            },
            {
                "startPosition": "ATwDAAADAQAA:4062",
                "endPosition": "ATwDAAADAQAA:4062",
                "creationTime": "2019-08-11T15:24:03.088000",
                "lastModificationTime": "2019-08-11T15:24:03.088000",
                "template": "0\ufffc0",
                "note": "This is my first  note in this book"
            },
            {
                "startPosition": "AWwDAACcAAAA:13111",
                "endPosition": "AWwDAACcAAAA:13111",
                "creationTime": "2019-08-11T15:24:03.079000",
                "lastModificationTime": "2019-08-11T15:24:03.079000",
                "template": "0\ufffc0",
                "note": "More notes"
            },
            {
                "startPosition": "ASIEAABwAAAA:42227",
                "endPosition": "ASIEAABwAAAA:42227",
                "creationTime": "2019-08-11T15:24:03.088000",
                "lastModificationTime": "2019-08-11T15:24:03.088000",
                "template": "0\ufffc0",
                "note": "A really long highlight"
            }
        ],
        "annotation.personal.bookmark": [
            {
                "startPosition": "AVoDAAAAAAAA:9430",
                "endPosition": "AVoDAAAAAAAA:9430",
                "creationTime": "2019-08-11T15:24:03.088000",
                "lastModificationTime": "2019-08-11T15:24:03.088000",
                "template": "0\ufffc0"
            },
            {
                "startPosition": "AUsDAAAAAAAA:6642",
                "endPosition": "AUsDAAAAAAAA:6642",
                "creationTime": "2019-08-11T15:24:03.088000",
                "lastModificationTime": "2019-08-11T15:24:03.088000",
                "template": "0\ufffc0"
            }
        ]
    },
    "ReaderMetrics": {
        "booklaunchedbefore": "true"
    },
    "erl": "AcgiAAA0AAAA:1206501"
}
Attached Files
File Type: zip krds-v1.zip (3.8 KB, 13 views)

Last edited by jhowell; 08-13-2019 at 07:14 PM. Reason: Update info
jhowell is online now   Reply With Quote
Old 08-12-2019, 06:31 PM   #2
j.p.s
Wizard
j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.
 
Posts: 2,249
Karma: 35799999
Join Date: Apr 2011
Device: pb360
Wow! Thanks!

I'll try kicking the tires when I get some time.
j.p.s is online now   Reply With Quote
Advert
Old 08-12-2019, 07:54 PM   #3
jhowell
Wizard
jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.
 
jhowell's Avatar
 
Posts: 2,915
Karma: 28068759
Join Date: Nov 2011
Location: Florida
Device: Oasis 2, Fire, iPad Air 2, Nexus 7
Quote:
Originally Posted by j.p.s View Post
I'll try kicking the tires when I get some time.
I hope that what I found out can be useful in your project.
jhowell is online now   Reply With Quote
Old 08-13-2019, 03:28 PM   #4
PoP
 curly᷂͓̫̙᷊̥̮̾ͯͤͭͬͦͨ ʎʌɹnɔ
PoP ought to be getting tired of karma fortunes by now.PoP ought to be getting tired of karma fortunes by now.PoP ought to be getting tired of karma fortunes by now.PoP ought to be getting tired of karma fortunes by now.PoP ought to be getting tired of karma fortunes by now.PoP ought to be getting tired of karma fortunes by now.PoP ought to be getting tired of karma fortunes by now.PoP ought to be getting tired of karma fortunes by now.PoP ought to be getting tired of karma fortunes by now.PoP ought to be getting tired of karma fortunes by now.PoP ought to be getting tired of karma fortunes by now.
 
PoP's Avatar
 
Posts: 2,615
Karma: 40180611
Join Date: Dec 2010
Location: ♁ ᴺ₄₅°₃₀' ᵂ₇₃°₃₇' ±₆₀"
Device: K3₃.₄.₂ PW3&4₅.₁₂.₁
Quote:
Originally Posted by jhowell View Post
...
the file extensions are .mbs and .mbp1 (for MOBI), .azw3f and .azw3r (for KF8), and .yjf and .yjr (for KFX)
...
Thanks for sharing. Also found that .pds and .pdt (for PDF) have that signature and decode similarly.


Spoiler:
.pds
Code:
{
    "font.prefs": {
        "typeface": "_INVALID_,und:bookerly",
        "lineSp": -1,
        "size": -1,
        "align": -1,
        "insetTop": 28,
        "insetLeft": 28,
        "insetBottom": 0,
        "insetRight": 28,
        "unknown1": -1,
        "bold": -1,
        "userSideloadableFont": "",
        "customFontIndex": -1,
        "mobi7SystemFont": "_INVALID_,und:bookerly",
        "mobi7RestoreFont": false,
        "readingPresetSelected": ""
    },
    "sync_lpr": false,
    "annotation.cache.object": {
        "annotation.personal.highlight": [
            {
                "startPosition": "1 6 51 1 255 177 53 17",
                "endPosition": "1 48 280 1 305 345 79 21",
                "creationTime": "2019-08-13T16:05:05.820000",
                "lastModificationTime": "2019-08-13T16:05:05.820000",
                "template": "0\ufffc0"
            },
            {
                "startPosition": "1 67 361 1 123 445 63 17",
                "endPosition": "1 108 9 1 450 1049 13 21",
                "creationTime": "2019-08-13T16:06:03.643000",
                "lastModificationTime": "2019-08-13T16:06:03.643000",
                "template": "0\ufffc0"
            }
        ],
        "annotation.personal.note": [
            {
                "startPosition": "1 48 280 1 305 345 79 21",
                "endPosition": "1 48 280 1 305 345 79 21",
                "creationTime": "2019-08-13T16:05:46.309000",
                "lastModificationTime": "2019-08-13T16:05:46.309000",
                "template": "0\ufffc0",
                "note": "Ingr\u00e9dients"
            },
            {
                "startPosition": "1 108 9 1 450 1049 13 21",
                "endPosition": "1 108 9 1 450 1049 13 21",
                "creationTime": "2019-08-13T16:06:23.148000",
                "lastModificationTime": "2019-08-13T16:06:23.148000",
                "template": "0\ufffc0",
                "note": "Recette"
            }
        ],
        "annotation.personal.bookmark": [
            {
                "startPosition": "1 0 0 0",
                "endPosition": "1 0 0 0",
                "creationTime": "2019-08-13T16:01:53.130000",
                "lastModificationTime": "2019-08-13T16:01:53.130000",
                "template": "0\ufffc0"
            }
        ]
    },
    "language.store": {
        "language": "fr",
        "unknown1": 0
    },
    "ReaderMetrics": {
        "booklaunchedbefore": "true"
    }
}
.pdt
Code:
{
    "fpr": {
        "position": "80 0 0 0",
        "time": null,
        "timeZoneOffset": null,
        "country": "",
        "device": ""
    },
    "page.history.store": [],
    "lpr": {
        "position": "1 0 0 0",
        "time": "2019-08-13T16:06:55.068000"
    }
}

Last edited by PoP; 08-13-2019 at 04:17 PM. Reason: add generated json
PoP is offline   Reply With Quote
Old 08-13-2019, 07:04 PM   #5
jhowell
Wizard
jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.
 
jhowell's Avatar
 
Posts: 2,915
Karma: 28068759
Join Date: Nov 2011
Location: Florida
Device: Oasis 2, Fire, iPad Air 2, Nexus 7
Quote:
Originally Posted by PoP View Post
Thanks for sharing. Also found that .pds and .pdt (for PDF) have that signature and decode similarly.
Thanks for the info.

I tested a Topaz (.azw1) file and it uses .tal and .tas files with the same type of content.

I will update the first post to add this information.
jhowell is online now   Reply With Quote
Advert
Old 08-14-2019, 07:36 AM   #6
shamanNS
Evangelist
shamanNS ought to be getting tired of karma fortunes by now.shamanNS ought to be getting tired of karma fortunes by now.shamanNS ought to be getting tired of karma fortunes by now.shamanNS ought to be getting tired of karma fortunes by now.shamanNS ought to be getting tired of karma fortunes by now.shamanNS ought to be getting tired of karma fortunes by now.shamanNS ought to be getting tired of karma fortunes by now.shamanNS ought to be getting tired of karma fortunes by now.shamanNS ought to be getting tired of karma fortunes by now.shamanNS ought to be getting tired of karma fortunes by now.shamanNS ought to be getting tired of karma fortunes by now.
 
Posts: 476
Karma: 1200116
Join Date: Feb 2010
Location: Serbia
Device: Kindle PW3[300dpi] , Kindle PW1
So, this script does not extract the actual text that was highlighted?
shamanNS is offline   Reply With Quote
Old 08-14-2019, 08:09 AM   #7
jhowell
Wizard
jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.
 
jhowell's Avatar
 
Posts: 2,915
Karma: 28068759
Join Date: Nov 2011
Location: Florida
Device: Oasis 2, Fire, iPad Air 2, Nexus 7
Quote:
Originally Posted by shamanNS View Post
So, this script does not extract the actual text that was highlighted?
That is correct. The script decodes whatever is in the files indicated in the first post of this thread. The reader application has no need to store the actual text separately from the book format file.

The linkage between the files that this program decodes and the book's content are fields labeled with "position" in the name. These are strings that identify where to find content within a book and are interpreted differently for each book format.

KF8 (azw3) format appears to be the simplest case. The position is a decimal number giving an offset within the raw HTML content of the book, as can be obtained using the kindleunpack software. See the work done by j.p.s for an example of how to make use of this information.

MOBI (azw) format is similar, but there appears to be additional information that I have not attempted to decode.

KFX uses two values separated by a colon. The first is a base64 encoding of the eid and offset, which are fields used internally by KFX to determine the location of content. The second is the actual position number, which in the case of KFX counts visible unicode characters instead of raw HTML bytes.

I have not looked into how position numbers are handled in the other formats that Kindle supports.

Last edited by jhowell; 08-14-2019 at 08:16 AM.
jhowell is online now   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Lost Kindle Data files Alastair_Lack Amazon Kindle 5 06-28-2013 03:31 PM
Where does the Android Kindle app store ebook files? bfollowell Amazon Kindle 3 03-12-2013 05:04 PM
Where does Reader for PC store files? redbees Sony Reader 12 06-27-2012 07:03 PM
Where does Connect Reader store ebook files 1490peter Sony Reader 1 08-12-2009 03:45 AM
Old RSS data on Sony Store JeffASonyReader Sony Reader 3 01-14-2008 06:20 PM


All times are GMT -4. The time now is 07:15 PM.


MobileRead.com is a privately owned, operated and funded community.