reverse engineering azw3r?

[blaenk]

I've been reading a book for months now and making important highlights as I go, and yesterday I realized that all highlights from the first 60% of the book were gone (I'm at around 80% now).

I loaded up the kindle and I see both the .azw3r file, which AFAIK contains the offsets for the highlights, but I also see an .azw3r.bad_file file alongside it. So somehow it became corrupted? I have no idea.

I opened them both in a hex editor and I definitely see data in both. I have no experience with reverse engineering, but I'm willing to put in the work if it means that I may have a chance to recover those highlights.

Preferably I'd love to merge both, but alternatively I'd love it if I could fix and use the old one since it should have way more highlights, then I can duplicate the book with a different name and give that one the newer .azw3r, then I can manually add the new highlights to the old one.

I guess I'm mainly curious if anyone has any idea what kind of format the file is in. I imagine it should be the same or similar to the other azw3 metadata files? Is it some serialization format similar to protocol buffers, BSON, or Thrift? Or is it some other custom format? And does anyone know the way in which the data is arranged? Are the highlights stored from earliest to latest in the book? Could there be a way to determine why the .bad_file one is considered to be a bad file, in the hopes that I may fix it?

I'm mainly looking for some direction to get started with this, as I've never done this sort of thing. Or if someone knows that it's hopeless then I won't bother

As an aside, the reason these highlights are important to me is that I read through the book making highlights as I go, then I go back and write up notes based on the highlights (i.e. the important bits). This allows me to read at a faster pace than I would if I had to writes notes as I went along. It has worked perfectly for dozens of books that I've read before, I just never imagined that the rug would be pulled from under me like this!

[geekmaster]

Hope this helps:

Originally Posted by https://wiki.mobileread.com/wiki/KF8

KF8 (also called AZW3) is basically a compiled ePub that has been compiled using a Palm database and Amazon's DRM scheme. It is targeting ePub 3 version support but existing ePub can also be used. The latest KindleGen software will create KF8. KF8 replaces MOBI but actually includes both a MOBI database and a KF8 database in the same file for backwards compatibility using older Amazon Kindle readers. ... The internal format has been decoded using kindle unpack. It reveals that the basic format remains as a PDB file similar in structure to MOBI. Generally a KF8 file may contain both a MOBI file at the beginning and the newer KF8 version of ePub later. This of course increased the size of the book file although there is some attempt to share resources such as images between the two objects. Kindle Unpack can be used to separate the two structures by building a traditional MOBI file and a KF8 file without the MOBI part (actually a small dummy structure remains) to make the file smaller. There are flags near the start of the database that can be used to identify the type of file. In spite of the fact that KF8 is targeted at ePub 3 source it still carries forward some HTML constructions that have been de-standardized for years and left over from MOBI's use. Amazon does publish the HTML statements and CSS3 statements that it will recognize. In addition KF8 has extended the ePub 3 format in incompatible ways to support its fixed layout option. All Kindle products beginning with the Kindle 3 can support KF8, although the Kindle 3 requires an update to do so.

Originally Posted by https://wiki.mobileread.com/wiki/KindleUnpack

Kindle unpack (originally called Mobi unpack), also known as Mobi decode, is a python script that creates MOBI or AZW (Amazon Kindle) source files from the compiled database. The filenames used in the source file are not necessarily the same as those that were originally used to create the database as this information is not preserved in the database but an unpacked set of files should be able to be used to recreate the same database using standard mobi or Kindle generating tools. For KF8 files and combined Mobipocket and KF8 files built by KindleGen, it also can produce separated mobipocket and KF8 files, and also the original source files if those are included in the eBook. In addition, for KF8 files it can produce an 'ePub', although if the HTML isn't compliant with ePub standards, the 'ePub' won't be either. For Amazon's .azw4 files, it will extract the PDF that's been wrapped up in Amazon's .azw4 file format. A Calibre plugin version of the scripts is available in this thread.

The trailing 'r' on the file extension means it is a "sidecar" (metadata) file, which in this case may contain your added notes. Here is a related mobileread discussion regarding sidecar (including azw3r) files: https://www.mobileread.com/forums/sho...d.php?t=262356

[blaenk]

Thanks! But the linked thread is one that I created, and unfortunately doesn't contain much information about the metadata file which contains the highlights (azw3r AFAIK).

As for the quotes, thanks as well, but I have a feeling that KF8/kindleunpack is independent of the highlights no? That is, the highlights are stored outside of the actual azw3 book, in the sdr folder's azw3r file.

[knc1]

Aren't the highlight's file just offset and length into the text of the book?

If that is the case, then you will have to unpack the book to reach that text content.
I.E: They will not be offset and length into the packed file.

[eschwartz]

I did find references for the older .mbp format (used for storing annotations data for MOBI books).
http://www.angelfire.com/ego2/idleloop/mbp_reader.html

It is possible you can get some idea how it works/worked from there.

As a general rule, lab126 is fond of their byte offsets.
But no one has really devoted time to exploring the wonders of the annotations files....

[geekmaster]

Here is a kindle highlights fetcher (with source code):

https://rubygems.org/gems/kindle-highlights

https://github.com/speric/kindle-highlights

It may have some code snippets to guide you in your "missing highlights" recovery efforts.

One note of interest regarding this tool:

Amazon will sometimes issue a CAPTCHA challenge when logging in to your Kindle account. If this happens when the gem attempts to log in to your Kindle account to retrieve your book list or highlights, you'll get a KindleHighlights::Client::CaptchaError ... There's no way to programmatically resolve this situation. The best solution I've found is to open a browser, visit the URL that the gem returns, log in to your Kindle account, and click around a bit. Then log out of your Kindle account and re-attempt to fetch your highlights via this gem.

[Yourcat]

You may ask Amazon whether they can fix the file. Hacking the format may take longer than re-reading the book. As you have backups of both files (and hopefully everything else) you could try to rename the bad file to .azw3r and hope that it can be read by your PW after a restart/reboot.