Troubleshooting Convert Metadata file .AZW3R into .MBP or "MyClippings.txt" into a Metadata file

[ilovejedd]

By the way, something I totally forgot about. You can skip the conversion via kindlegen by using EPUB directly and letting Amazon do the conversion.

1. Rename EPUB to JPG
2. Use the renamed JPG for "Send to Kindle"

It looks like Amazon uses DOC compression (c1) for files sent this way.

[LittleBeard]

Originally Posted by jhowell

That is not surprising. The files you replaced contain a cache the annotations, but the master copy is still on Amazon's server. There is no way to force those cached annotations to be sent to the server. Only newly generated annotations are sent back to Amazon.

Ahh that is really useful to understand.
I presume, that is the case with any metadata file sideloaded to the kindle vs. being created on the device itself, right?
So it even if it could be done to generate a .azw3r/-f equivalent file for a .mobi book (.mbp, etc) then the corresponding annotations and highlights (sideloaded to the kindle) would not be uploaded to amazons servers unless they were created or edited on the kindle (or other kindle app or device). Is that correct?

Originally Posted by jhowell

That is also to be expected. Annotations locate the precise point in the book using a "position" number that is based on the exact HTML content of the file produced by kindlegen. The e-book source file recovered via kindleunpack will not match the original source file exactly, leading to this sort of minor discrepancy.

Okay. Thanks a lot for the clarification! So, whenever a book is converted from one format to another it will generate these kind of discrepancies…

[LittleBeard]

Originally Posted by ilovejedd

Finally in front of an actual PC.
[LIST][*]I don't actually use the standard "Unpack AZW3" option. Instead, I do the ff:

• Select all AZW3 files I want to process
• Use "KF8 to ePub..." option in KindleUnpack (adds the EPUB for all selected books with AZW3 format directly to the library)
• Save to disk -> Save single format to disk... -> EPUB (I use a flat structure as Save to disk template for saving to the kindlegen folder)

Sweet! Yes that's way quicker and more efficient. I like it!

Originally Posted by ilovejedd

• Sorry, I don't use Kindle Previewer so can't help with that.
  
• Tested this and it works for converting a bunch of EPUBs in the same folder. Make sure kindlegen.exe is either in the same folder as your EPUB files or in Windows' PATH (still on Win7 so dunno how to add this on Win10).
  

  Command:
  forfiles /m *.epub /c "cmd /c kindlegen.exe <options> @file"
  
  
  Options:
    -c0                   No compression
    -c1                   Standard DOC compression (fast: default)
    -c2                   Huffdic compression (smallest file sizes but very slow)
    -dont_append_source   Don't include the EPUB source file in the MOBI output

Yes putting the KindleGen.exe into the same folder as the .epub really makes things easy!

Originally Posted by ilovejedd

[*]Yep. Dual-format in this case means you have two formats inside that single MOBI file: KF6 (old MOBI) and KF8 (AZW3). That's part of the reason the file sizes are so big. In your case, the generated file also includes the original EPUB input so that's actually 3 files: KF6, KF8 and EPUB.

Here are the results for some compression tests I previously did for kindlegen if you wish to slim those down.

EPUB source
~700 KB: 1.78 MB uncompressed, ~275,000 words, 65KB cover image

forfiles /m *.epub /c "cmd /c kindlegen.exe -c1 @file"
2.73 MB, 2-3 seconds processing time on Core i5-3450S 2.80 GHz

forfiles /m *.epub /c "cmd /c kindlegen.exe -c1 -dont_append_source @file"
2.05 MB, 2-3 seconds

forfiles /m *.epub /c "cmd /c kindlegen.exe -c2 @file"
2.04 MB, 1 minute 28 seconds

forfiles /m *.epub /c "cmd /c kindlegen.exe -c2 -dont_append_source @file"
1.36 MB, 1 minute 28 seconds

Originally Posted by j.p.s

Since kindlegen makes a dual mobi (KF7 and KF8 format), the output file is usually
considerably larger than the input EPUB. But it also by default includes a copy of the input EPUB file. Note the in post #14 the -dont_append_source option is listed. Using that option omits the copy of the EPUB from the output dual mobi. The output will likely still be larger than the EPUB, but at least it won't include a full copy of it.

Using the "-dont_append_source" command really makes a good difference!
Its bigger only by margins compared to the original .epub file =)

Originally Posted by ilovejedd

By the way, I just tested this on my devices.

1. Created annotations on an unregistered PW3.
2. Copied the sidecar files from PW3 to a registered PW4 (with WhisperSync enabled and already had a copy of the AZW3 ebook/doc from Kindle cloud archive)
3. Renamed azw3r/f files to add the user id.
4. Fiddled with annotations on PW4 (select annotation, move the end marker a little and immediately return it to its original position)
5. Closed ebook and manually ran sync.
6. Checked on registered PW2 and Kindle for iOS (iPad) and the annotations I fiddled with were uploaded.

Note, you do need to fiddle with every single annotation to get it uploaded. Since the highlights are off by a bit, fixing them by hand on the Kindle would upload all the annotations you've fixed.

Conversion and upload of dual MOBI is pretty simple so if I were you, I'd just do that first followed by copy/rename of the old sidecar files. Then just fix/upload the highlights at leisure.

Ingenious. I followed your instructions and it works really well.
Just synched my 1st book across devices including annotations and highlights!

Does anyone know if it's possible to move the positions of annotations?
After I've moved the highlights to their correct place, the postitions of the annotations float in mid text with thier offset...

Cheers again =)

[j.p.s]

Originally Posted by LittleBeard

But first let me share what I did exactly to see if I did everything correctly and also for anyone else who want's to follow suit and needs a little more detail...

I forgot to mentions that it is very helpful when people do this, and it makes it much easier to help.

Thanks!

[j.p.s]

Originally Posted by LittleBeard

Does anyone know if it's possible to move the positions of annotations?
After I've moved the highlights to their correct place, the postitions of the annotations float in mid text with thier offset...

I don't follow. Can you post a screenshot?

[jhowell]

Originally Posted by LittleBeard

I presume, that is the case with any metadata file sideloaded to the kindle vs. being created on the device itself, right?

So it even if it could be done to generate a .azw3r/-f equivalent file for a .mobi book (.mbp, etc) then the corresponding annotations and highlights (sideloaded to the kindle) would not be uploaded to amazons servers unless they were created or edited on the kindle (or other kindle app or device). Is that correct?

Correct.

[LittleBeard]

Originally Posted by j.p.s

I forgot to mentions that it is very helpful when people do this, and it makes it much easier to help.

Thanks!

Awesome. A win-win I would say!
Thanks for the feedback =)

[LittleBeard]

Originally Posted by j.p.s

I don't follow. Can you post a screenshot?

https://ibb.co/wwwgr0y


It's a screenshot made on my Kindle for Android. But it's the same on my Kindle PW2...

It's the position of an old annotation. But since I've moved the highlight, the position of the annotation doesn't move. It stays in the old, now "wrong" position.

[LittleBeard]

Originally Posted by ilovejedd

By the way, something I totally forgot about. You can skip the conversion via kindlegen by using EPUB directly and letting Amazon do the conversion.

1. Rename EPUB to JPG
2. Use the renamed JPG for "Send to Kindle"

It looks like Amazon uses DOC compression (c1) for files sent this way.

Nice. I've just tried that. And it works good aswell =)

[LittleBeard]

I want to add and edit quite a few annotations and highlights for a non-Amazon book (in my personal Documents thats allready synched between my Android and Kindle devices) more quickly and conveniently.
Is it possible to use Kindle for PC? On the Kindle it takes quite a long time. As on my Kindle for Android too!

Is it correct that as of today that is not possible?
https://www.amazonforum.com/s/questi...-windows-10-pc

Because doing this:
https://www.howtogeek.com/howto/1316...kindle-for-pc/
won't synchronize any annotations and highlights I create on Kindle for PC to other devices...

So many question ^^
But I'm really glad to have found this forum =)

[LittleBeard]

Originally Posted by LittleBeard

I want to add and edit quite a few annotations and highlights for a non-Amazon book (in my personal Documents thats allready synched between my Android and Kindle devices) more quickly and conveniently.
Is it possible to use Kindle for PC? On the Kindle it takes quite a long time. As on my Kindle for Android too!

Is it correct that as of today that is not possible?
https://www.amazonforum.com/s/questi...-windows-10-pc

Because doing this:
https://www.howtogeek.com/howto/1316...kindle-for-pc/
won't synchronize any annotations and highlights I create on Kindle for PC to other devices...

So many question ^^
But I'm really glad to have found this forum =)

Just a small update conserning the above:
I found a neat work-around that works well for annotating personal (non-Amazon) documents on a Windows computer:

I installed an Android emulator (BlueStacks) on my Windows device. Installing the Kindle App on that Android emulation makes it easy to copy, paste, search, find and edit annotations, highlights, bookmarks, etc within the personal document that are synced between Kindle devices!