Amazon removed books from my Kindle

[pdurrant]

Originally Posted by Hollow Man

This is how I chose to solve it, but I suspect most here are antsy about having a copy of their non Amazon books being loaded on Amazon's servers.

I have switched my buying to Kobo, but I'm still using Send To Kindle for my reading devices. I have vast numbers of ebooks bought from other ebook stores, and I've never worried about using the Amazon servers.

Of the 5400+ book in my calibre database, only 1500 or so have been bought from Amazon, although that is my largest number from one source. The now defunct Fictionwise is second with around 1200, followed by Baen at 900, with Kobo coming a distant 4th at 243.

[hondabf]

Regardless of the fact that there are people who misuse the EBOK tag, or use hacks to have the books in PDOC tags or if AZW3 and KFX are proprietary formats, there will still be clueless people who will use Calibre to sideload books (not knowing the EBOK tag will be set by default) because it has always been the go-to program to sideload books locally, only to find the book deleting bug still present some time later.

I know ignorance is not an excuse to be upset at anyone, but it still surprises me that kovid has neither given an option to set the cdetype tags or at the very least, warned the users about it, considering the large history between Calibre and Kindles.

KFX enthusiasts with the PDOC option enabled and older Kindle users are the only ones who are safe from this bug that keeps occurring.

[BionicGecko]

Originally Posted by hondabf

Regardless of the fact that there are people who misuse the EBOK tag, or use hacks to have the books in PDOC tags or if AZW3 and KFX are proprietary formats, there will still be clueless people who will use Calibre to sideload books (not knowing the EBOK tag will be set by default) because it has always been the go-to program to sideload books locally, only to find the book deleting bug still present some time later.

This is a very fair point; I don’t normally use Calibre to manage my Kindle library so I tend to forget that the content type is set to EBOK by default by Calibre. It is indeed not realistic to expect everyone using Calibre casually to be fully aware of what is done behind the scenes when they send their book to their device.

[axu2]

Originally Posted by jhowell

I have a Kindle Oasis 2 running 5.16.2.1.1 firmware that has been offline for a couple of months so I thought I would use it for a test of e-book deletion. The device only had books on it that were purchased from Amazon or delivered using send-to-kindle.

I created some additional e-books for testing and sideloaded them via USB. This included at least three books each in AZW3 (KF8) and KFX formats. Some were marked as a Kindle purchase (CDETYPE=EBOK), personal document (CDETYPE=PDOC), and not specified (no CDETYPE metadata).

I saved a copy of the documents folder of the device, turned off airplane mode, and synced it with Amazon. The result was the deletion of all of the books I had USB sideloaded with CDETYPE=EBOK. No other books were deleted from the device.

A system log created using ;dm showed the same number of the following messages as the number of books that were deleted:
240429:145014 cvm[10279]: I ReaderSDKImpl:Information::Processing remove item
240429:145014 cvm[10279]: I ReaderSDKImpl:Information::Successfully deleted content

To me this appears to be evidence of the intentional deletion of books that are marked internally as having come from Amazon for which there is no associated record in the user's account.

That info about EBOK/PDOC/empty cdetype was exactly what I was looking for, thanks! For fun I loaded books with a cdetype of TEST and the book didn't appear in kindle at all.

I wonder if using the cdetype=EBSP (ebook sample) is useful at all. Or the blank one

[endriu]

Originally Posted by jhowell

A system log created using ;dm showed the same number of the following messages as the number of books that were deleted:
240429:145014 cvm[10279]: I ReaderSDKImpl:Information::Processing remove item
240429:145014 cvm[10279]: I ReaderSDKImpl:Information::Successfully deleted content

Given this information, is there any way to develop a patch that would block this behavior for jailbroken devices? As a software developer I could try to help or contribute to a solution, if given some initial guidance on how to approach this

[j.p.s]

Kindle firmwares were written with the assumption that no one else is manipulating CDETYPE. amazon also made the customer hostile decision to not show cover thumbnails for personal documents. The latter led to the former happening.

Kindle syncing firmware is closed source. Changing the syncing firmware would require decompiling obfuscated java and changing it.

There are several known "fixes".

1. Don't muck with CDETYPE.

2. Keep wifi always off

3. Keep wifi frequently on.

4. Only use "send to kindle" or similar to load books.

[Sirtel]

Originally Posted by j.p.s

Kindle firmwares were written with the assumption that no one else is manipulating CDETYPE. amazon also made the customer hostile decision to not show cover thumbnails for personal documents. The latter led to the former happening.

Kindle syncing firmware is closed source. Changing the syncing firmware would require decompiling obfuscated java and changing it.

There are several known "fixes".

1. Don't muck with CDETYPE.

The problem with the first fix is that Calibre sets the CDETYPE automatically to EBOK when converting to AZW3. Kovid has refused to change this behavior. Most folks who are not regular MR visitors have no idea that the CDETYPE of their sideloaded books has been changed by converting them to AZW3.

[j.p.s]

Originally Posted by Sirtel

The problem with the first fix is that Calibre sets the CDETYPE automatically to EBOK when converting to AZW3. Kovid has refused to change this behavior. Most folks who are not regular MR visitors have no idea that the CDETYPE of their sideloaded books has been changed by converting them to AZW3.

Well, the word fixes was deliberately in quotes. All 4 have problems. if Mr. Goyal won't add the option, what makes anyone think amazon will cave, or even know this is happening?

I guess #5 is: "Don't use kindles."

[Sirtel]

Originally Posted by j.p.s

I guess #5 is: "Don't use kindles."

That was Kovid's reply to the posters who asked him to change this behavior.

And no, I don't think Amazon will change it either at this point. Then again, who knows...

[Timboli]

Originally Posted by hondabf

On Calibre, you can modify the PDOC tag on AZW3 books, the thing is that there are no plugins for it yet.

Also, why not mentioning the tip that you can set KFX books as PDOC? Anyways, in the mean time you can hack the QualityCheck plugin to set the PDOC tag on AZW3 books. Here's a quick guide for it.

If you're not using Calibre you might also want to take a look to the Java Mobi Metadata Editor.

Thanks for sharing all that.

That guide could have been simplified a little, by telling you just to extract the check_fix.py file, edit that, and then copy it back to the QualityCheck.zip file, overwriting the original check_fix.py file. Of course you would backup the QualityCheck.zip file first.

Anyway, worked a treat for me ... not that I am aware of any of my ebooks on my Kindles having been deleted thus far.

I'm thinking of creating a BAT file, as I've done in the past, to swap in or out either version of the QualityCheck.zip file. So just using the modified version at need, but having the original be the default.

Not seen or used the Java Mobi Metadata Editor program before, and it likewise worked a treat, where I used it to manually change EBOK to PDOC, in an AZW3 file, which was simple as.

Cheers

[Timboli]

Originally Posted by j.p.s

Well, the word fixes was deliberately in quotes. All 4 have problems. if Mr. Goyal won't add the option, what makes anyone think amazon will cave, or even know this is happening?

I guess #5 is: "Don't use kindles."

#6 is: Change EBOK to PDOC using a suitable method.

See my previous post for that method ... two available.

The Java Mobi Metadata Editor program is the simplest if you can execute JAR files. No need to use calibre.

P.S. I have also seen where the default value is set as PDOC when converting with calibre to MOBI. So that is another option.

[endriu]

Originally Posted by j.p.s

Changing the syncing firmware would require decompiling obfuscated java and changing it.

I realize it would involve reverse engineering, but things like that have been done before with some of the kindle hacks and mods, haven't they?

[rupor]

Originally Posted by Timboli

#6 is: Change EBOK to PDOC using a suitable method.

See my previous post for that method ... two available.

If you like me prefer USB sideloading I guess I have #7:

Keep local directory with your books and use sync2kindle. When you turn on WiFi and your books are deleted simply repeat something like "s2k usb --ignore-device-removals". It will copy back to device whatever Amazon deleted without screwing your last position and such. And for EBOK books will also extract and send to device proper thumbnails keeping everything in order. This was one of the main reasons I created this tool - jailbreaking and decompiling firmware when company does not want you to mess with it is too much of a hustle really and messing with additional editing of the metadata did not sound that appealing either.

[j.p.s]

Originally Posted by endriu

I realize it would involve reverse engineering, but things like that have been done before with some of the kindle hacks and mods, haven't they?

Yes, some have done that.

[Timboli]

Originally Posted by rupor

If you like me prefer USB sideloading I guess I have #7:

Keep local directory with your books and use sync2kindle. When you turn on WiFi and your books are deleted simply repeat something like "s2k usb --ignore-device-removals". It will copy back to device whatever Amazon deleted without screwing your last position and such. And for EBOK books will also extract and send to device proper thumbnails keeping everything in order. This was one of the main reasons I created this tool - jailbreaking and decompiling firmware when company does not want you to mess with it is too much of a hustle really and messing with additional editing of the metadata did not sound that appealing either.

Very interesting. Thanks for sharing.

I do however belong to the school of 'prevention is better than cure'.

This is especially so, considering how both my Kindle devices are full, and I would not be wanting to have any removals and the restoring of such, which could potentially be a lot.

I will keep your program in mind though.