3G: How to download highlights?

[feklee]

Is there a way to download highlights from a Kindle 3G? (yes, that old, but still good)

They're stored in a .han file. However, that's not really useful. The text is missing:

{"payload":{"key":"B085YDW8TX","type":"EBOK","records":[{"endPosition":"132668","startPosition":"132556","lastModificationTime":"2021-07-22 20:35:01.535","type":"kindle.highlight"},{"text":"test","endPosition":"132668","startPosition":"132668","lastModificationTime":"2021-07-22 20:35:01.535","type":"kindle.note"},{"endPosition":"137165","startPosition":"137149","lastModificationTime":"2021-07-22 20:35:01.535","type":"kindle.highlight"},{"endPosition":"146709","startPosition":"146562","lastModificationTime":"2021-07-22 20:35:01.535","type":"kindle.highlight"},{"endPosition":"149543","startPosition":"149440","lastModificationTime":"2021-07-22 20:35:01.535","type":"kindle.highlight"},{"endPosition":"151942","startPosition":"151771","lastModificationTime":"2021-07-22 20:35:01.535","type":"kindle.highlight"},{"endPosition":"167827","startPosition":"167631","lastModificationTime":"2021-07-22 20:35:01.535","type":"kindle.highlight"},{"endPosition":"173261","startPosition":"173093","lastModificationTime":"2021-07-22 20:35:01.535","type":"kindle.highlight"},{"endPosition":"197978","startPosition":"197869","lastModificationTime":"2021-07-22 20:35:01.535","type":"kindle.highlight"},{"endPosition":"20...

[tomsem]

If your books are not side-loaded (either purchased from Amazon or a Send To Kindle ‘personal document’), and you enable sync (default), then the highlights are in the cloud. You can use any of the Kindle apps (Fire, iOS, Android) to export them. For purchased books, there’s a website where you can view them and do copy/paste etc.

Newer Kindles will export notes and highlights directly (but only for purchased books).

There is also ‘my clippings.txt’, and tools that will extract stuff from there.

And I think there is way tool that could get the annotations from sidecar files for the older format. I don’t know the current status of it.

[j.p.s]

Originally Posted by tomsem

And I think there is way tool that could get the annotations from sidecar files for the older format. I don’t know the current status of it.

The tools to extract the information have been around for a couple of years, but no one has provided a GUI to drive them.

https://www.mobileread.com/forums/sh...d.php?t=321811

https://www.mobileread.com/forums/sh...d.php?t=322172

[feklee]

Originally Posted by j.p.s

https://www.mobileread.com/forums/sh...d.php?t=321811

https://www.mobileread.com/forums/sh...d.php?t=322172

Thanks! I don’t think however that these tools are compatible with the file layout on the Kindle 3G:

China The Bubble that Never Pops-asin_B085YDW8TX-type_EBOK-v_0.apnx
China The Bubble that Never Pops-asin_B085YDW8TX-type_EBOK-v_0.azw3
China The Bubble that Never Pops-asin_B085YDW8TX-type_EBOK-v_0.ea
China The Bubble that Never Pops-asin_B085YDW8TX-type_EBOK-v_0.han
China The Bubble that Never Pops-asin_B085YDW8TX-type_EBOK-v_0.phl

This gave no output:

azw3r -i 'China The Bubble that Never Pops-asin_B085YDW8TX-type_EBOK-v_0.azw3'

And this gave me a KindleReaderDataStore signature is incorrect exception:

python krds.py 'China The Bubble that Never Pops-asin_B085YDW8TX-type_EBOK-v_0.azw3'

I also tried the tools on the .han file and other files, no success. Apparently, the format has changed.

[j.p.s]

Both of the tools are meant to be run on .azw3r files. All the azw3 books I have store highlight and note information in azw3r files that are in some strange java related binary format. I had never heard of .han files, but it looks like they are already in a usable format, perhaps JSON. (I am not familiar enough with JSON to tell and do not have time to look into it.)

If kindleunpack works on your azw3 files you can use it to get the raw XHTML file of the book. The startPositions and endPositions in your .han file are very likely byte (or maybe character) offsets in the raw XHTML file of the highlighted text. A link to kindleunpack information is given in post #7 in the KRDS thread linked to above.

[feklee]

Thanks for the suggestions! I realized that Kindle 3G is not the accurate description of my device. It’s a Kindle Keyboard 3G.

[feklee]

Originally Posted by j.p.s

I had never heard of .han files, but it looks like they are already in a usable format, perhaps JSON.

Yes, JSON.

If kindleunpack works on your azw3 files you can use it to get the raw XHTML file of the book. The startPositions and endPositions in your .han file are very likely byte (or maybe character) offsets in the raw XHTML file of the highlighted text.

Not right away, though, but maybe after stripping the markup.

Interestingly I can get a list of all annotations in Duokan, which I have installed for dual-boot on my device. It’s a 20 page document, and maybe I’ll just scan that with a flatbed scanner. Kind of dumb, but way quicker than trying to reverse engineer that format.

I don’t think I want to annotate with the Kindle again. This is all a pain. Fortunately, it seems to be easy to de-drm books, and then I can use another reader.

[j.p.s]

Amazon startPositions and endPositions include the markup. Stripping the markup would render them useless.

[feklee]

Originally Posted by j.p.s

Amazon startPositions and endPositions include the markup. Stripping the markup would render them useless.

I had a second look. First of all, the .rawml file in the mobi8 directory seems to be all ASCII. Non-ASCII characters are encoded using HTML entities. For verification:

$ wc -m  book.rawml
793974 book.rawml
$ wc -c  book.rawml
793974 book.rawml

So I picked a highlight from the .han file:

{"endPosition":"316113","startPosition":"315821","lastModificationTime":"2021-07-22 20:43:27.262","type":"kindle.highlight"}

To get the text I needed to add an offet of 15 to the positions (spaces added by me, to prevent phpbb from expanding entities):

$ cut -z -c$((315821 + 15))-$((316113 + 15)) book.rawml
The solution wasn&# 8217;t just another round of stimulus. After all, &# 8220;with global growth weak, using stimulus to use up excess capacity is like preparing food for two when there&# 8217;s only one guest; they could eat as much as they could and it still wouldn&# 8217;t all be gone.&# 8221;

This matches what I could find in the list of annotations shown in Duokan:

The solution wasn’t just another round of stimulus. After all, “with global growth weak, using stimulus to use up excess capacity is like preparing food for two when there’s only one guest; they could eat as much as they could and it still wouldn’t all be gone.”

But all this is unnecessarily complicated. The easiest solution is to just download documents/My Clippings.txt from the device. It has the annotations of all books in it.