Troubleshooting Convert Metadata file .AZW3R into .MBP or "MyClippings.txt" into a Metadata file

[LittleBeard]

Hi Everyone and what an awesome forum!

I've spent the last few days browsing this forum (and the web) for any clues I could find to solve the following problem:

In the last 6 years I've mainly used the USB connection to transfer non-Amazon Books with Calibre to my Kindle PW 2. Doing so I've always converted them into .AZW3.
Doing this Whisper Sync was never in action or used (since books were not sent by email).

Now I've lost my kindle, but luckily always made backups of myClippings.txt and uploaded that to https://www.clippings.io/ to edit annotations for book summeries, using Evernote.
I've also made a backup a few months ago of the "documents" folder from my kindle with all the Metadata files in the .SDR folders of each book.

Sidenote: During this search I found out it would have been way smarter to have sent ALL the books over the years via Email to my Kindle Email adress to get access to Whispersync and Sync all books across devices.

My Question:
Is there a way to convert a Kindle metadata file (in my case .AZW3R) into .MBP or .MBP1 file?
Or extract my annotations and notes from "MyClippings.txt" and convert it into a metadata file (for example .mbp) for each individual book?

The reason is that I would love to send all my books to my used/newly baught Kindle PW 2 via Email in .MOBI format and then copy the .SDR folders with the converted .MBP or MBP1 files manually onto my Kindle.

Would that even work?
Any ideas, solutions or other (possibly simpler) solutions would be more than welcome!

[mergen3107]

I don't know about sending by email, but for transferring books and their respective highlights to able to see them all on another Kindle, can be done as described in this post.

[LittleBeard]

Thank you very much mergen3107.

Your indicated post was indeed quite insightful insofar as I had no idea that that "random" number represented the ID.
I noticed that since I'm using the same amazon account and have changed the new device name to the old device name (don't know if that makes a difference...) the ID seems to be identical and I don't even have to change it on the new device.
Just coppying the books (.azw3) and .SDR folders (.with azw3r and .azw3f extensions) works fine and annotations and highlights are shown on kindle.

Unfortunately, that didn't quite solve my problem.
My goal is to synchronise my books in the cloud on the new device (so I can edit books on all devices and won't risk loosing annotations, etc again).
For that I need to send them via email and will end up with either a .mobi or a .azw book file (depending on conversion), but so far have worked out no way of getting a .azw3 file when sending it to the device. (If that was possible, that would be the easiest solution for me!)

That is the core of my problem, since I cannot use the backed-up .SDR folders of my .azw3 books. The extensions are just different (for example for .mobi the metadate files are .mbs, .mbp1, etc) and can't be replaced by .azw3r or .azw3f metadata files.

That's why in my naivety I thought it would make sense to convert the .azw3r and .azw3f files into .mbs and .mbp1

Any suggestions?

[mergen3107]

AFAIK, azw3 files themselves lack good cloud support. For example, you cannot send them via Send to Kindle.
Also, since these format are proprietary and Amazon insists on that, we cannot generate them, unfortunately.
If you are concerned about saving these highlights and notes to be able to read them in future, you can save them with Kindle Mate software, for example.

[LittleBeard]

Wow thanks for the quick reply and clarification again.

I've got Kindle Mate installed. Kindle Mate does not provide the solution I'm looking for, but provides a good method for backing up the content - as I understand it.

But since I want to have the saved highlights and annotations in the cloud together with the book or better said to view them in the book itself (for better context, etc) - I'm faced with the dilemma of having used and saved my books so far in azw3 format (included generating the correlative azw3r and azw3f metadata files) which I cannot use to restore my annotations and highlights if I send them to the device as mobi or azw.

It looks like I'll have to select my most worked-on and important books, send them via email to kindle as .mobi, open www.clippings.io with my saved annotations and highlights and restore the clippings manually... If it's not tooo time consuming.

But in the future, I'll definitely send new books via .mobi for sure and simply use WhisperSync across all devices!

Thanks again for the prompt replies and trying to understand what it is I'm looking for ^^

[mergen3107]

You are welcome! I wish there was a better solution for you.

For now it is only one way: we can only read and extract highlights from .azw3r, but cannot create such files. For completeness of your investigation, look at this thread.

[ilovejedd]

You can try doing some roundtrip conversions for your AZW3 files to get dual MOBI on Amazon cloud which should deliver the AZW3 version to your device (caveat, no covers for AZW3).

1. If you don't have it yet, install KindleUnpack plugin to Calibre then use KindleUnpack to extract EPUB from AZW3. You can do this in batch.
2. Use kindlegen to convert from the extracted EPUB to dual-format MOBI. You can automate this step via batch script. I'm on iPad right now but if you're on Windows, I can probably cook up a batch file for this when I'm back on a PC.
   https://www.amazon.com/gp/feature.ht...cId=1000765211
   Note:
   • We're NOT using Calibre's built-in conversion for steps 1-2 to minimize changes made to the books. Hopefully, this will result in the AZW3 Kindle Cloud versions that match your existing annotations more closely.
3. Upload dual-format MOBI created by kindlegen via Email/Send to Kindle. The Send to Kindle for PC/Mac drag and drop window can send the ebook directly to multiple devices along with the Kindle cloud archive.
   https://www.amazon.com/gp/sendtokindle/pc
   https://www.amazon.com/gp/sendtokindle/mac
4. Rename the .sdr folders and .azw3r and .azw3f files to match the filenames of the downloaded ebook and append the id at the end if needed.
   I believe the template is:
   Title_bookid.azw3
   Title_bookid.sdr\
   Title_bookiduserid.azw3f
   Title_bookiduserid.azw3r
   Notes:
   • As you have noticed, the id that mergen3107 noted is actually a user id based on your Amazon account. In case it's missing, you can just use batch renaming software to append this text to the filenames for all devices linked to your Amazon account.
   • The renaming to match the downloaded filename will likely take quite a bit of elbow grease. Still, likely not as time-consuming as creating annotations from scratch. You might also need to review the annotations to get them uploaded to Kindle Cloud and to make minor adjustments.

[mergen3107]

"is actually a user id based on your Amazon account"
Interesting! Thank you

[LittleBeard]

Originally Posted by ilovejedd

You can try doing some roundtrip conversions for your AZW3 files to get dual MOBI on Amazon cloud which should deliver the AZW3 version to your device (caveat, no covers for AZW3).

1. If you don't have it yet, install KindleUnpack plugin to Calibre then use KindleUnpack to extract EPUB from AZW3. You can do this in batch.
2. Use kindlegen to convert from the extracted EPUB to dual-format MOBI. You can automate this step via batch script. I'm on iPad right now but if you're on Windows, I can probably cook up a batch file for this when I'm back on a PC.
   https://www.amazon.com/gp/feature.ht...cId=1000765211
   Note:
   • We're NOT using Calibre's built-in conversion for steps 1-2 to minimize changes made to the books. Hopefully, this will result in the AZW3 Kindle Cloud versions that match your existing annotations more closely.
3. Upload dual-format MOBI created by kindlegen via Email/Send to Kindle. The Send to Kindle for PC/Mac drag and drop window can send the ebook directly to multiple devices along with the Kindle cloud archive.
   https://www.amazon.com/gp/sendtokindle/pc
   https://www.amazon.com/gp/sendtokindle/mac
4. Rename the .sdr folders and .azw3r and .azw3f files to match the filenames of the downloaded ebook and append the id at the end if needed.
   I believe the template is:
   Title_bookid.azw3
   Title_bookid.sdr\
   Title_bookiduserid.azw3f
   Title_bookiduserid.azw3r
   Notes:
   • As you have noticed, the id that mergen3107 noted is actually a user id based on your Amazon account. In case it's missing, you can just use batch renaming software to append this text to the filenames for all devices linked to your Amazon account.
   • The renaming to match the downloaded filename will likely take quite a bit of elbow grease. Still, likely not as time-consuming as creating annotations from scratch. You might also need to review the annotations to get them uploaded to Kindle Cloud and to make minor adjustments.

Yeahhh if your idea works ilovejedd, then I'll jump up and down out of joy!
Thanks so much for the detailed description already!

Unfortunately I already have my problems with step 1.
I've installed the "KindleUnpack - The Plugin (0.82.1) by DiapDealer" via Calibre. I'll have to look into what to do next...

Concerning step 2:
Sadly, I've never really done any coding and get by with the occasional command prompt on windows.
I can't find an easy way to open/use Kindlegen right now. I'll look into it later tomorrow when I've slept!

Looking forward to get this working!
Sorry for being so slow - this is not my area of strength!

[tomsem]

IMO, it is hard to beat the Export Notes feature that is built into Kindle. If these are annotations for a book not from Amazon, you need only use Kindle app for Fire, iOS, or Android, which do this (as long as it is stored in cloud).

If you side-load exclusively then none of this works of course.

[LittleBeard]

Originally Posted by ilovejedd

You can try doing some roundtrip conversions for your AZW3 files to get dual MOBI on Amazon cloud which should deliver the AZW3 version to your device (caveat, no covers for AZW3).

1. If you don't have it yet, install KindleUnpack plugin to Calibre then use KindleUnpack to extract EPUB from AZW3. You can do this in batch.
2. Use kindlegen to convert from the extracted EPUB to dual-format MOBI. You can automate this step via batch script. I'm on iPad right now but if you're on Windows, I can probably cook up a batch file for this when I'm back on a PC.
   https://www.amazon.com/gp/feature.ht...cId=1000765211
   Note:
   • We're NOT using Calibre's built-in conversion for steps 1-2 to minimize changes made to the books. Hopefully, this will result in the AZW3 Kindle Cloud versions that match your existing annotations more closely.
3. Upload dual-format MOBI created by kindlegen via Email/Send to Kindle. The Send to Kindle for PC/Mac drag and drop window can send the ebook directly to multiple devices along with the Kindle cloud archive.
   https://www.amazon.com/gp/sendtokindle/pc
   https://www.amazon.com/gp/sendtokindle/mac
4. Rename the .sdr folders and .azw3r and .azw3f files to match the filenames of the downloaded ebook and append the id at the end if needed.
   I believe the template is:
   Title_bookid.azw3
   Title_bookid.sdr\
   Title_bookiduserid.azw3f
   Title_bookiduserid.azw3r
   Notes:
   • As you have noticed, the id that mergen3107 noted is actually a user id based on your Amazon account. In case it's missing, you can just use batch renaming software to append this text to the filenames for all devices linked to your Amazon account.
   • The renaming to match the downloaded filename will likely take quite a bit of elbow grease. Still, likely not as time-consuming as creating annotations from scratch. You might also need to review the annotations to get them uploaded to Kindle Cloud and to make minor adjustments.

All right @ilovejedd and all interested:
I've managed to complete steps listed above with a half successful outcome - see Result at the end of the post.
But first let me share what I did exactly to see if I did everything correctly and also for anyone else who want's to follow suit and needs a little more detail...

1. I've installed the "Kindle Unpack - The Plugin (0.82.1) by DiapDealer" via Calibre.
   It's available by searching "KindleUnpack" in the plugins on Calibre.
   A yellow exploding triangle appears in the main menu bar.
   I've configured the output option via "Customize Plugin": epub output version: ePub2, then selected a book had on my previous Kindle in .azw3 format, clicked on "KindleUnpack", selected "AZW3" and "Unpack AZW3".
   Opening the folder, it was saved in I got my epub file.
   
   
2. Following the instructions for KindleGen I firstly tried doing it with the Kindle Previewer because I couldn’t get the KindleGen.exe working with the command prompt:
   https://kindlegen.s3.amazonaws.com/Readme.txt:
   
   - Download the Kindle Previewer from http://www.amazon.com/kindleformat/kindlepreviewer
   - Once the download is finished, install the Kindle Previewer
   - If you need further instruction on how to install Kindle Previewer, please refer http://kindlepreviewer.s3.amazonaws.com/UserGuide.pdf
   - Once the install is complete, launch Kindle Previewer
   - Click on "Open book" link in Kindle Previewer
   - Select the EPUB/HTML/OPF to convert to Kindle ebook
   - Follow the instructions to convert the book and preview
   - The converted eBook with extension ".mobi" can be obtained from the folder Compiled-filename from the same folder as the source HTML/EPUB.
   
   Then I did manage to do it with KindleGen directly with the help of this thread:
   https://www.mobileread.com/forums/sh...d.php?t=169427
   with the following command line in command prompt:
   kindlegen "ebook_name.epub" -c1 -o "ebook_name.mobi"
   
   The output file is a .mobi almost double in size to the .epub file.
   
   
3. Then I uploaded that .mobi file with Send to Kindle for PC to my Kindle.
   My question is if that generated file is the mentioned "dual-format" MOBI?
   Since there is no other mobi file... I guess it must be?!
   
   
4. I opened the book on my Kindle and did a few experimental annotations and highlights (also to generate the .SDR folder).
   Then I changed the names of the two files in the Kindle directory:
   Title_bookiduserid.azw3f
   Title_bookiduserid.azw3r
   to match the BOOK ID and USER ID.
   

Side note:
"WhisperSync" is enabled on all devices now.
Also, the option "archive personal documents" is enabled on my Amazon account.

Result:
The annotations and highlights from my previous files are displayed on my kindle after I've copied the files over and disconnected the kindle from the PC.

Issue 1:
However, annotations and highlights do not seem to be synced to my other devices (Kindle on Android) for example.
The book itself appears in my "devices" on Amazon and on my Kindle for Android app though. The hightlights are not displayed!

When I make annotations on this Android device the annotations and highlights on my kindle disappear and are replaced by the annotations and highlights made on the android device. I can keep adding annotations and highlights on the Kindle, which will in turn be synced to the android device. So, from then on WhisperSync seems to work. But only with the newly added, not the originally file-based ones.

When I connect the Kindle to my PC again via USB the .SDR folders content is down to 1kB each for the azw3r and azw3f file. So, it's been replaced.


Issue 2:
The highlights displayed on the kindle after using the old azw3f and -r files are off by about 82 characters. Though this seems to differ from book to book. I might be able to amend that using Kindle Previewer, I guess. But it's difficult to figure out the exact number of characters (even within ones book highlights), because it varies. Probably due to the fact that I just copied the effected texts over to MS Word to count characters and paragraphs, titles, etc might be counted differently...


It's good progress. But I'm a little disheartened if this will work in the end.
I've fiddled around disabling WhisperSync until the book is safely on the device and then syncing it when the annotations and highlights are on the device too. But it makes no difference. The new highlights replace the old (wanted) highlights.

Kind regards and thanks again for all of your patience and happy for any other ideas =)

[jhowell]

Originally Posted by LittleBeard

Issue 1:
However, annotations and highlights do not seem to be synced to my other devices (Kindle on Android) for example.

That is not surprising. The files you replaced contain a cache the annotations, but the master copy is still on Amazon's server. There is no way to force those cached annotations to be sent to the server. Only newly generated annotations are sent back to Amazon.

Originally Posted by LittleBeard

Issue 2:
The highlights displayed on the kindle after using the old azw3f and -r files are off by about 82 characters. Though this seems to differ from book to book.

That is also to be expected. Annotations locate the precise point in the book using a "position" number that is based on the exact HTML content of the file produced by kindlegen. The e-book source file recovered via kindleunpack will not match the original source file exactly, leading to this sort of minor discrepancy.

[ilovejedd]

Originally Posted by LittleBeard

Result:
The annotations and highlights from my previous files are displayed on my kindle after I've copied the files over and disconnected the kindle from the PC.

Issue 1:
However, annotations and highlights do not seem to be synced to my other devices (Kindle on Android) for example.
The book itself appears in my "devices" on Amazon and on my Kindle for Android app though. The hightlights are not displayed!

When I make annotations on this Android device the annotations and highlights on my kindle disappear and are replaced by the annotations and highlights made on the android device. I can keep adding annotations and highlights on the Kindle, which will in turn be synced to the android device. So, from then on WhisperSync seems to work. But only with the newly added, not the originally file-based ones.

When I connect the Kindle to my PC again via USB the .SDR folders content is down to 1kB each for the azw3r and azw3f file. So, it's been replaced.

After you have copied and renamed the old .azw3r/f files, try selecting/fixing all the old highlights on the Kindle first before opening the book on other devices and see if that syncs the old highlights to the Kindle cloud.

Issue 2:
The highlights displayed on the kindle after using the old azw3f and -r files are off by about 82 characters. Though this seems to differ from book to book. I might be able to amend that using Kindle Previewer, I guess. But it's difficult to figure out the exact number of characters (even within ones book highlights), because it varies. Probably due to the fact that I just copied the effected texts over to MS Word to count characters and paragraphs, titles, etc might be counted differently...

As @jhowell mentioned, this is to be expected. We used kindlegen to try to minimize this discrepancy (Calibre conversion will likely be even more off). Unfortunately, we can't eliminate it entirely.

[ilovejedd]

Originally Posted by LittleBeard

Spoiler:

1. I've installed the "Kindle Unpack - The Plugin (0.82.1) by DiapDealer" via Calibre.
   It's available by searching "KindleUnpack" in the plugins on Calibre.
   A yellow exploding triangle appears in the main menu bar.
   I've configured the output option via "Customize Plugin": epub output version: ePub2, then selected a book had on my previous Kindle in .azw3 format, clicked on "KindleUnpack", selected "AZW3" and "Unpack AZW3".
   Opening the folder, it was saved in I got my epub file.
   
   
2. Following the instructions for KindleGen I firstly tried doing it with the Kindle Previewer because I couldn’t get the KindleGen.exe working with the command prompt:
   https://kindlegen.s3.amazonaws.com/Readme.txt:
   
   - Download the Kindle Previewer from http://www.amazon.com/kindleformat/kindlepreviewer
   - Once the download is finished, install the Kindle Previewer
   - If you need further instruction on how to install Kindle Previewer, please refer http://kindlepreviewer.s3.amazonaws.com/UserGuide.pdf
   - Once the install is complete, launch Kindle Previewer
   - Click on "Open book" link in Kindle Previewer
   - Select the EPUB/HTML/OPF to convert to Kindle ebook
   - Follow the instructions to convert the book and preview
   - The converted eBook with extension ".mobi" can be obtained from the folder Compiled-filename from the same folder as the source HTML/EPUB.
   
   Then I did manage to do it with KindleGen directly with the help of this thread:
   https://www.mobileread.com/forums/sh...d.php?t=169427
   with the following command line in command prompt:
   kindlegen "ebook_name.epub" -c1 -o "ebook_name.mobi"
   
   The output file is a .mobi almost double in size to the .epub file.
   
   
3. Then I uploaded that .mobi file with Send to Kindle for PC to my Kindle.
   My question is if that generated file is the mentioned "dual-format" MOBI?
   Since there is no other mobi file... I guess it must be?!

Finally in front of an actual PC.

1. I don't actually use the standard "Unpack AZW3" option. Instead, I do the ff:
   • Select all AZW3 files I want to process
   • Use "KF8 to ePub..." option in KindleUnpack (adds the EPUB for all selected books with AZW3 format directly to the library)
   • Save to disk -> Save single format to disk... -> EPUB (I use a flat structure as Save to disk template for saving to the kindlegen folder)
2. Sorry, I don't use Kindle Previewer so can't help with that.
   
   Tested this and it works for converting a bunch of EPUBs in the same folder. Make sure kindlegen.exe is either in the same folder as your EPUB files or in Windows' PATH (still on Win7 so dunno how to add this on Win10).
   

   Command:
   forfiles /m *.epub /c "cmd /c kindlegen.exe <options> @file"
   
   
   Options:
     -c0                   No compression
     -c1                   Standard DOC compression (fast: default)
     -c2                   Huffdic compression (smallest file sizes but very slow)
     -dont_append_source   Don't include the EPUB source file in the MOBI output

3. Yep. Dual-format in this case means you have two formats inside that single MOBI file: KF6 (old MOBI) and KF8 (AZW3). That's part of the reason the file sizes are so big. In your case, the generated file also includes the original EPUB input so that's actually 3 files: KF6, KF8 and EPUB.
   
   Here are the results for some compression tests I previously did for kindlegen if you wish to slim those down.
   
   

   EPUB source
   ~700 KB: 1.78 MB uncompressed, ~275,000 words, 65KB cover image
   
   forfiles /m *.epub /c "cmd /c kindlegen.exe -c1 @file"
   2.73 MB, 2-3 seconds processing time on Core i5-3450S 2.80 GHz
   
   forfiles /m *.epub /c "cmd /c kindlegen.exe -c1 -dont_append_source @file"
   2.05 MB, 2-3 seconds
   
   forfiles /m *.epub /c "cmd /c kindlegen.exe -c2 @file"
   2.04 MB, 1 minute 28 seconds
   
   forfiles /m *.epub /c "cmd /c kindlegen.exe -c2 -dont_append_source @file"
   1.36 MB, 1 minute 28 seconds

By the way, I just tested this on my devices.

1. Created annotations on an unregistered PW3.
2. Copied the sidecar files from PW3 to a registered PW4 (with WhisperSync enabled and already had a copy of the AZW3 ebook/doc from Kindle cloud archive)
3. Renamed azw3r/f files to add the user id.
4. Fiddled with annotations on PW4 (select annotation, move the end marker a little and immediately return it to its original position)
5. Closed ebook and manually ran sync.
6. Checked on registered PW2 and Kindle for iOS (iPad) and the annotations I fiddled with were uploaded.

Note, you do need to fiddle with every single annotation to get it uploaded. Since the highlights are off by a bit, fixing them by hand on the Kindle would upload all the annotations you've fixed.

Conversion and upload of dual MOBI is pretty simple so if I were you, I'd just do that first followed by copy/rename of the old sidecar files. Then just fix/upload the highlights at leisure.

[j.p.s]

Originally Posted by LittleBeard

Then I did manage to do it with KindleGen directly with the help of this thread:
https://www.mobileread.com/forums/sh...d.php?t=169427
with the following command line in command prompt:
kindlegen "ebook_name.epub" -c1 -o "ebook_name.mobi"

The output file is a .mobi almost double in size to the .epub file.

Since kindlegen makes a dual mobi (KF7 and KF8 format), the output file is usually
considerably larger than the input EPUB. But it also by default includes a copy of the input EPUB file. Note the in post #14 the -dont_append_source option is listed. Using that option omits the copy of the EPUB from the output dual mobi. The output will likely still be larger than the EPUB, but at least it won't include a full copy of it.