Calibre Sideloaded and Notes from Amazon Cloud

[nynaevelan]

Ok:

I am not sure if I should be asking this question here or in the Calibre forum. So I will ask here and if this is the wrong place I will try the Calibre forum next.

I use Calibre to manage all my ebooks including the ones I've purchased from Amazon. All of my edited books are recognized by Amazon so there is no problem there. I side load my Amazon books via Calibre. My question is this: How do I get my notes/highlights/bookmarks which are in the Amazon cloud for these sideloaded books to show up? I've tried syncing but nothing seems to bring them over. So far the only way I've been able to get them is if I download the book directly from the Amazon Cloud, which I do not want to do because I lose my plugboard and metadata adjustments that way. Is there any way to accomplish this without directly downloading them from Amazon?

Escwartz: You are my resident expert, is this possible??

I swear every time I get one thing solved, another problem pops up.

[crich70]

I think notes and highlights are linked to the DRM that Amazon has built into their books. That's how you can read a book you buy from them 1st on your kindle and then pick up where you left off using Kindle for pc or the reader app on your cell phone as well. I'm not sure you can access notes etc. that you have made in ebooks if the books are side loaded rather than downloaded from the cloud.

[eschwartz]

I am fairly sure it doesn't rely on the DRM itself, as you can have books without DRM. Plus I use DeDRMed only books (with plugboards applied) and it still gets synced. But it probably needs you to use the file Amazon provides since they don't use the same conversion calibre does, plus it will for sure need to have a matching ASIN -- which you are already doing, right?

Conversion adds in all kinds of changes, many in an unpredictable manner. One thing I know is that synced annotations data relies on location, which is calculated in some obscure manner that takes formatting into account. So when calibre converts and organizes the book's css formatting, it changes the precise location calculations. It is possible that throws off your Kindle enough that Amazon cannot recognize the book as being the same book (even if, for the purposes of collections, it is recognized) causing it to fail.

Or perhaps your Kindle recognizes the book when it comes to collections, but not when matching the book for the sake of collections.

This is totally a guess, though, and good luck getting Amazon to confirm anything!

Someday I may reverse-engineer the whole of the Kindle and answer your question. (And many others, no doubt!)

I cannot remember why you needed to convert the books, but as far as the metadata/plugboards applied by calibre, it is not necessary to convert the book first, as calibre should apply those changes when sending Amazon's (imported) files to your Kindle, and according to my experiences, neither the Kindle nor Amazon should notice the difference, but of course if you need to edit the books to fix typos, as many people do, you need to convert in order to edit. Once the book is converted, it is no longer Amazon's book, and we had to use the ASIN trick just to get collections working, butI have no idea where we go from there to getting any other similarities recognized.

[eschwartz]

Confirmed that DeDRMed (but not converted) books will in fact still record annotations if I happen to sync while logged into my sister's account, who does not in fact actually own the books in question.

I don't want to assume that Amazon has cleverly *fixed* this with the 5.4.3/4 firmware, (it synced with KT 5.3.2.1) but for all I know, there may actually be some difference there. I can't imagine why though. Just thought I'd mention it. I guess.

[nynaevelan]

Originally Posted by eschwartz

Confirmed that DeDRMed (but not converted) books will in fact still record annotations if I happen to sync while logged into my sister's account, who does not in fact actually own the books in question.

I don't want to assume that Amazon has cleverly *fixed* this with the 5.4.3/4 firmware, (it synced with KT 5.3.2.1) but for all I know, there may actually be some difference there. I can't imagine why though. Just thought I'd mention it. I guess.

I could have confirmed this for you because all my books are DeDRMed, and all are recognized by Amazon as being "their" books. But no worries, I have gone to the trouble of downloading them all from Amazon again. Now I am have to have a convo with myself to figure out if I am going to keep the Amazon version on the device, or put the sideloaded version back on and rename the accompanying files for Amazon so that they have their bookmarks.

One may ask, why do I bother? Simply, I want the existing bookmarks/notes in the files... Good thing I am home bored...

[eschwartz]

Originally Posted by nynaevelan

I could have confirmed this for you because all my books are DeDRMed, and all are recognized by Amazon as being "their" books. But no worries, I have gone to the trouble of downloading them all from Amazon again. Now I am have to have a convo with myself to figure out if I am going to keep the Amazon version on the device, or put the sideloaded version back on and rename the accompanying files for Amazon so that they have their bookmarks.

One may ask, why do I bother? Simply, I want the existing bookmarks/notes in the files... Good thing I am home bored...

I was merely observing that annotations are unaffected by the change. So much so that even the account which doesn't own the book still has the notes.

If you like, I can guide you as to how I deal with sending to device so annotations match up.

[nynaevelan]

Originally Posted by eschwartz

I was merely observing that annotations are unaffected by the change. So much so that even the account which doesn't own the book still has the notes.

If you like, I can guide you as to how I deal with sending to device so annotations match up.

Yes please.

[GoSharks]

Originally Posted by nynaevelan

Yes please.

Ditto!

[eschwartz]

OK, so the hardest part of this is getting the data into calibre in the first place.

I have a custom column named "#file", titled "Original Filename". In it, I store the filename that the Amazon book had when I added it from my Kindle, minus the file extension. It will be in the form of {book-title}_{ASIN}. I add this while I clean up the book on adding it, one here and one there, but adding many in bulk will not be a pleasant task, I imagine. (Sorry. ) I've been doing it since the beginning, though. (It was how I initially stumbled over MobileRead, I need help with "plugboards, templates, and custom columns". This is before I then became a maven myself. )

I use a custom template for sending to device, (limited to the kindle2 device driver, since other devices don't need this,) which looks like this:

program: 
ifempty(
	field('#file'),
	template(
		'{author_sort:sublist(0,1,&)}/{title} - {authors:sublist(0,1,&)}'
	)
)

It uses the contents of custom column "file" as the savepath, if available. If not, (for instance, it isn't originally from Amazon, so I left it blank) it falls back on the main template, which I actually tweaked to only use the first author.

[nynaevelan]

Ok, I do not think that is going to help me, I have wayyyyyy too many books to try to go back and find the original name. It would probably be easier to just copy the files over and rename them there.

[nynaevelan]

When you send your files to the device, are you also sending the mbp files? If so, is Calibre putting them in the sdr folders??

[eschwartz]

Originally Posted by nynaevelan

When you send your files to the device, are you also sending the mbp files? If so, is Calibre putting them in the sdr folders??

No, but the .sdr will download from Amazon. (Or you can back them up. I also use a "kindlesync" script so I don't lose any of them once they've been on my Kindle before.)

Someday I will write a plugin that will let calibre understand those files and put them into the right place, though.

[nynaevelan]

What's a kindlesync?

[eschwartz]

Originally Posted by nynaevelan

What's a kindlesync?

It's a script I wrote, which syncs my Kindle Drive.

It uses rsync to sync the contents of the documents folder plus "system/thumbnails" when changes are made, and if anything has been deleted, to sync it back. There are exclusion rules to control what gets sent, it's also how I delete EndActions.

[nynaevelan]

Originally Posted by eschwartz

It's a script I wrote, which syncs my Kindle Drive.

It uses rsync to sync the contents of the documents folder plus "system/thumbnails" when changes are made, and if anything has been deleted, to sync it back. There are exclusion rules to control what gets sent, it's also how I delete EndActions.

hmmm that sounds interesting, any chance you will wanna share?? Is this syncing within Calibre or to your hard drive??