Associate personal documents to Kindle store books

[atava]

Hi all, my first post. I've registered expressly in these forums to know from some of you if what I need/want is possible.
I've searched the forums briefly but I didn't find what I was looking for and I think it is a very specific question anyway.

Currently I'm using the Amazon Kindle app for Android. I like it and possibly I will purchase more books from the Kindle store (and maybe also a Kindle itself) and stick to it for my future reading.
Now, in the past few years I've bought many drm-ed books from other places (e.g. Google Books and some other online stores) and I tried to add them to the Kindle cloud by putting them into Calibre, de-drmize them, convert them to mobi and then upload them as personal documents with the @kindle emails.
Despite some minor conversion defects (which I'd like to see solved) - such as margins/alignment getting screwed up at times - the books do show as personal documents in my Kindle app and the reading point is synced as well as highlights/notes.
So far, so good.

Now, this is the most important part to me, the highlights and the notes.
They get stored and synchronized in the Amazon cloud fine, but I've seen that the uploaded books aren't "linked" to their counterparts in the Kindle store, despite the ISBN and ASIN tags being set to the correct values (I've already read about ASIN, and experimented with mobi tags but with no success).
What I want is that if I upload a no-drm book as a personal document it gets linked to its corresponding Kindle store page, so that if I want to purchase it again from the Kindle store and then delete the document the attached highlights and notes are still stored in the purchased book.

Example:
- I upload Dune by Frank Herbert as a personal document (a previous purchase) by removing the drm in Calibre and then sending it to the @kindle email as mobi. I start reading it and add some highlights/notes.
- This Dune by Frank Herbert document gets associated to the Kindle store page for the book (I mean the book with that ISBN, that particular edition). One day I find that Amazon offers a very strong discount for the book, and I'd like to purchase the book again so not to have it as a personal document anymore but as a proper Amazon book.
- I purchase the book. Now I've got two entries for Dune by Frank Herbert in the app (or device): one is the personal document (the uploaded book), one is the newly-purchased book. If I open the new book - this is what I'd like to see - I find in it all the highlights and notes that I added to the personal document (and this because they are the same book).

All these problems may be summed up with a single question perhaps:
How does Amazon distinguish books when it comes to highlights/notes?
Does it use an ISBN (I'm quite sure it does not)? Does it use an ASIN (I don't think it does either)? Does it use a freshly-created id for the purchase? If so, I think I'm "doomed" and there's no way to achieve what I'd like to achieve here.

Thanks. I hope I made myself clear enough.

(And merry Christmas everybody)

[latepaul]

I don't think you can do this. Even if you could how would you get the highlights and notes to match locations?

[atava]

Well, ideally the textual contents of the two files would be the same (being the same ebook), so locations wouldn't change.

If I could take a look at the internals of how Amazon store highlights/notes I could already answer many of these questions myself.

(edit)
I've seen that Kindle devices create a MyClippings.txt file within their file system, but I don't know if this is where highlights are actually stored or a mere exports thereof.
(No isbn/asin/id in there, so I'd opt for the latter.)

[jhowell]

There are several problems with what you wish to do.

In most cases Amazon identifies books using a pair of values: CDE Type ("EBOK" for purchased books or "PDOC" for personal documents) and ASIN (10 characters for purchased books or 32 characters for personal documents). It doesn't matter what you set in your MOBI file before you email them. The values will be replaced by Amazon.

Amazon identifies positions within the book for notes and highlights with offset numbers that are very specific to each e-book produced. It is almost certain that the internal positions for books sold by Amazon will not match exactly with the same book purchased from another source and converted using calibre.

There are several ways to export notes and highlights for a book, usually with only approximate positions, called "locations" by Amazon. I don't know of any way to do the reverse and import this data. Some work toward understanding how this data is stored has been done for e-ink Kindle devices, but I haven't heard of anyone working out the internal file formats used for the Android app.

[atava]

Thank you for the feedback.

Yes, these were more or less my suppositions too, althought regarding locations I still hoped that something could work being the text the same (I thought that offsets could be applied to the actual text deprived of any custom data).

If I manage to get the "association" part with the store to work (using the values you mention, of which I already read here and here) I think I could still give this a try with some inexpensive book of those I already own from the Play Store (Google).
If the PDOC tag is added back in the emailing process I think I'm stuck though, as I don't own a Kindle and I can't "send directly" to any such device.

[DaleDe]

Originally Posted by atava

Thank you for the feedback.

If the PDOC tag is added back in the emailing process I think I'm stuck though, as I don't own a Kindle and I can't "send directly" to any such device.

You don't need a Kindle to have a Kindle App. They are available for PC, Mac, iOS, Android and are free downloads.

Dale

[atava]

Originally Posted by DaleDe

You don't need a Kindle to have a Kindle App. They are available for PC, Mac, iOS, Android and are free downloads.

Dale

Hi DaleDe. The problem was not that of having something Kindle-like with which to read mobi and awz files (I've been using the Android and PC versions of the app for months now), it was that of being able to send such files directly to the device (in my case, the phone) bypassing the send-to-mail step (becase this adds them as personal documents and not books).

For those interested (and reading), I managed to do that eventually. I saw that I could simply copy the files to the directory the Android app uses and then the app detects them as books (not personal documents). With the correct ASIN metadata tag they are also linked to their corresponding Kindle Store page.

So, the next step was trying to see whether what I wanted to achieve was possible.
I copied an already-purchased book (in this case, from Google Books) to the device and I applied some notes/highlights to it. Then I went to its Kindle Store page and I bought it again from Amazon, deleting the book and redownloading it (this, to be sure that I had the Amazon copy on the device and not the custom file anymore). Finally I opened this book to check if my previously applied notes/highlights (which I had added to the Google converted copy) were synced and preserved in this.
I did my tries many times and in many fashions. Without detailing the steps taken here, I think I can state that as a rule notes/highlights are not retained on switching to the Amazon "true" copy, and this for a simple reason (which was already foreseen above): locations aren't the same between the two files, at least in the case I tried (there were slight variations in one of the first chapters and this compromised all subsequent parts of the file).
This apparently was not due to Calibre's conversion process from epub to mobi/awz, but to the two stores (Google and Amazon) delivering two different copies for the same ebook (perhaps from two different updates by the publisher).

For some other books locations might indeed match, but I think this is too unreliable a base to start doing what I wanted to do in the first place, i.e. consolidating all my favorite books in the Kindle cloud by purchasing them again on Amazon whenever a discount pops up, meanwhile uploading my Google Books files. (Ultimately, I wanted to this because I like the Kindle app and Kindle synchronization more than other e-readers.)

It's a pity actually, because I found that the files where notes/highlights/last position are stored for each book (which on Windows have the mbp extension) are written in a very readable (and parsable) format.
Many of the tries I did involved copying the relevant sections of such files to the new ones. Doing thus I did succeed in passing the previously-made notes and highlights to the book purchased in Amazon, but - as I said - the highlights' locations translated to different parts of the book compared with the other copy, and this because of differences in location values between the two ebooks.

There's certainly room for plugins here (or software in general), something that would update the old location values to the new ones (taking raw text of the highlights/notes as a reference).
But I admit that my use case appear to be too much of a niche one to be of interest to anyone.

[DaleDe]

Amazon puts its own ID on files it supplies so even if the source is the same they can tell a Google from one of their own or a Calibre one. Of course books you purchase from Amazon is delivered and downloadable for their apps as well as their books. I use iOS and it retrieves books easily.

[Hitch]

Originally Posted by DaleDe

Amazon puts its own ID on files it supplies so even if the source is the same they can tell a Google from one of their own or a Calibre one. Of course books you purchase from Amazon is delivered and downloadable for their apps as well as their books. I use iOS and it retrieves books easily.

Not to mention--the HTML won't be the same, between the two files. Even if they are both put out by the same publisher, the likelihood is, the HTML/XHTML/CSS won't be the same. Therefore, the "locations" etc., will all be different.

Sorry--I don't see any way to do what you want. Nor do I see a demand for a plugin, other than your very specific and unique need. Of course, I could be wrong--if you write one, you never know, people might love it.

Hitch