Amazon removed books from my Kindle

[DNSB]

Originally Posted by Sirtel

It may be a calibre issue, but it doesn't mean it's ok for Amazon to just delete random files from users' devices without any action on the user's part.
All I did was to replace EBOK with PDOC in check_fix.py in the plugin archive. I'm not sure that's a method everyone would want to use, as it replaces a feature (fix ASIN for Kindle Fire) with its opposite. Ideally the fix would be added as an additional feature, not replacing an existing feature, but I'm no coder and that would be beyond my scant skill set.

Perhaps replacing the check for EBOK in check_mobi.py with a check for PDOC?

[Sirtel]

Originally Posted by DNSB

Perhaps replacing the check for EBOK in check_mobi.py with a check for PDOC?

Yes, but there must also be a way to change it, not only check.

[DiapDealer]

In case anyone missed it the first time around:

https://github.com/SolaProject/mobi_...v-file#example

is a pure python, one-file module that can be used to check/change kindlebook metadata. Including EXTH[501].

It's also available to be installed with pip as mobi-header.

[Sirtel]

Originally Posted by DiapDealer

In case anyone missed it the first time around:

https://github.com/SolaProject/mobi_...v-file#example

is a pure python, one-file module that can be used to check/change kindlebook metadata. Including EXTH[501].

It's also available to be installed with pip as mobi-header.

But you can't use it with many files in bulk like the Quality Check plugin, or can you? That was what I was looking for. It would be awfully time-consuming to change hundreds or even thousands of files one by one.

[DiapDealer]

Originally Posted by Sirtel

But you can't use it with many files in bulk like the Quality Check plugin, or can you? That was what I was looking for. It would be awfully time-consuming to change hundreds or even thousands of files one by one.

Sure you can. Someone would have to put together a calibre user interface plugin that could be run on all Kindlebook formats of books selected in the library. But all the heavy lifting of editing the metadata would be done by the mobi_header module, so it shouldn't be a herculean task by any means.

[DNSB]

The check_fix.py change allows changing it. The change in check_mobi.py allows for choosing files that do not have PDOC as the cdetype.

Basically any instances in config.py, check_fix.py and check_mobi.py of EBOK (upper case) was changed to PDOC which made the dialogs make more sense.

[haertig]

Originally Posted by DNSB

Perhaps replacing the check for EBOK in check_mobi.py with a check for PDOC?

Since the length of the text string "EBOK" is the same length as the text string "PDOC", replacement could be as simple as this (on a Linux computer):

sed -e 's/EBOK/PDOC/' -i.backup 'filename.azw3'

The "sed" command on Linux works fine on binary files for things like this. But "sed" on Unix, Windows or some other OS may not handle binary files appropriately. A wise proverb: "An assumption is the first step towards a screw-up". So don't assume this will work on other OS'es, even if they have the "sed" command. You have been warned!

I do not know the structure of an AZW3 file, and it may be perfectly find to alter the length of text like this when doing a replacement. But I can't say that for sure, since I know nothing about AZW3 files and any checksums (or whatever) they may or may not contain.

If you are more daring and don't want to create a backup copy of the original file before editing, you can replace the "-i.backup" part of the command with simply "-i". This would be fine if you keep current and solid backups of your Calibre library.

You could write a simple "for loop" script around the "sed" command to find and cycle through all the azw3 files you want to modify.

You may want to add some bulletproofness to the script and make sure that "EBOK" only appears once in the file before you go modifying it. If it appears more than once, you only want to do the replacement on the appearance in the metadata part of the file (which is probably the first occurrence, but you'd have to verify that).

The way to check it the text string appears multiple times is:

strings 'filename.azw3' | grep EBOK | wc -l

Note that if you use "-i.backup" it will create a backup copy of every file before it gets modified. So you will need twice the amount of storage space to allow two copies of each file (at least until you remove the *.backup copies after you've determined all is well and good).

[odamizu]

Originally Posted by Sirtel

The problem is, when you convert something to AZW3, Calibre sets the EBOK tag automatically and there's no way around it.

Originally Posted by odamizu

That strikes me as a calibre issue, not an Amazon problem.

Originally Posted by Sirtel

It may be a calibre issue, but it doesn't mean it's ok for Amazon to just delete random files from users' devices without any action on the user's part.

I don’t think it’s random at all. Though it occurs to me that users who always sideload via USB, always manage their ebooks/docs manually on their Kindle, and deliberately avoid connecting to WiFi are understandably unfamiliar with Kindle’s normal syncing process, so when they do connect to WiFi and books suddenly disappear, it can seem random, outrageous, alarming and/or nefarious.

I always connect to WiFi and manage content online (including using Send-to-Kindle to transfer all my personal documents), so when books appear and disappear whenever I turn on my Kindle or Kindle app it’s totally normal to me as part of Kindle’s normal syncing process. If I delete a book from my Kindle Library on one device, I expect it to disappear from all my other devices when I turn them on. If I log into my Amazon account and go to my Content Library or Devices to send or remove ebooks to specific devices or delete ebooks altogether from my Kindle Library, I expect those books to be added or deleted from my devices as soon as I turn them on.

I strongly suspect that in order to sync, Amazon maintains some sort of "master list” that records all Amazon-supplied ebooks (cdetype EBOK), all user-supplied personal documents (cdetype PDOC), what ebooks/documents the user has archived to their Content Library, and what ebooks/documents have been downloaded to each respective device.

I suspect every time a user connects to WiFi, Amazon synchronizes the device to the master list (and vice versa), which can mean automatically adding or deleting material from the device. If there is a new PDOC on the device that isn't on the master list, Amazon appears to assume it was transferred via USB and adds it to the master list. But if there is a new EBOK on the device that doesn't have a matching EBOK on the master list, Amazon appears to identify it as an error and deletes it. By "matching EBOK" I mean a file identical to a version supplied by Amazon and NOT a diddled version (to use DiapDealer's adjective, which is my new favorite word).

The bottom line seems to be: If you're sideloading personal documents (including ebook purchases that you’ve diddled with, even if the diddling is slight), you have two choices: either 1) never turn on WiFi, or 2) make sure it has cdetype PDOC so that Amazon knows it’s a personal document and will leave it alone. If you use a “fake” EBOK tag, you can’t blame Amazon for treating it like an EBOK.

The fact that calibre automatically uses a “fake” EBOK tag when converting ebooks to AZW3 is a calibre issue, which seems a not insurmountable problem for a willing developer.

The fact that users expect Amazon to leave their “fake” EBOKs alone seems, quite frankly, unreasonable, since Amazon provides the PDOC tag specifically for personal documents, including diddled Amazon purchases, which they leave alone.

[Aleron Ives]

Originally Posted by odamizu

The fact that calibre automatically uses a “fake” EBOK tag when converting ebooks to AZW3 is a calibre issue, which seems a not insurmountable problem for a willing developer.

The fact that users expect Amazon to leave their “fake” EBOKs alone seems, quite frankly, unreasonable, since Amazon provides the PDOC tag specifically for personal documents, including diddled Amazon purchases, which they leave alone.

This certainly seems like the bottom line to me. If Calibre is purposefully setting a tag that causes Amazon to apply syncing behaviour -- which includes book deletions in some cases as a normal and correct part of syncing -- then the solution is for Calibre to not set that tag (or at least to give you an option of whether you want the tag to be set).

This strikes me as more of a cleanup procedure than any kind of security measure, i.e. when you connect the Wi-Fi after a long break, Amazon sees that you have non-sideloaded (according to the EBOK tag, which sideloaded books should not have) books on your device that are not present on your other devices, so Amazon cleans up your device by removing the "outdated" files for you, as a normal part of syncing to ensure all your Kindles have the same books on them.

[Quoth]

There should not be PDOC and EBOK.

But it's irrelevant. Amazon should not ever delete content.

Their sync is broken and has been for years if you have an old mobi only kindle and azw3 Kindle on same account. The Amazon / Kindle management of series and collection is broken.

It's a mess created by Amazon.

[Sirtel]

I certainly don't agree that deleting content automatically, without any input from the user, is somehow ok, no matter the intention. But yeah, it seems we must solve this problem ourselves, as it's unlikely Amazon will fix it any time soon, if ever.

[Quoth]

Originally Posted by Quoth

There should not be PDOC and EBOK.

To clarify:
The way Amazon implements Personal vs non-personal is wrong, inc crippling covers etc. No-one else does this.
IMO the best reason to by an ereader, and the reason we bought them, is to read Public domain ebooks unavailable on paper. It's a bonus that it can be cheaper and more convenient to read bought novels as ebooks rather than on paper.

Often non-fiction or comics / graphic novels, text books and PDFs are better on a 10" to 14" real tablet.

But this is only one thing Amazon is doing wrong.

Solutions:

1. Never use Whispernet, but Download and Transfer for content bought off Amazon.
2. Never Sync. It's quicker to use search and find a phrase for position on another device. Amazon offers sync to track you and monetise your reading.
3. Never use KU. Buy ebooks. Better for you and the author. KDP Select cheats.
4. Never use Send To Kindle (App, email or web). It's giving Amazon a copy. It's less private, secure and envirnomentally poorer/

So don't turn on WiFi. Yes, all of this is a big problem for PDFs/Notes on a Scribe. I nearly bought one, but after reading Tomsem's experience and how it works I never will unless Amazon changes.

Complain to Amazon. Meanwhile only sideload, only buy Amazon content and remove the DRM using your Kindle serial number. Make backups.

[jlinear]

Funny how Amazon deletes my "diddled" Amazon purchased Calibre sideloaded content when I have absolutely no such problems using my "diddled" Amazon purchased Calibre sideloaded content on my Kobo devices.

[odamizu]

Originally Posted by Sirtel

I certainly don't agree that deleting content automatically, without any input from the user, is somehow ok, no matter the intention ...

Well, an argument could be made that users shouldn't violate the Kindle Store Terms of Use by stripping DRM, diddling with Kindle ebooks, and sideloading diddled Kindle ebooks with a fake cdetype EBOK tag.

Originally Posted by Quoth

The way Amazon implements Personal vs non-personal is wrong, inc crippling covers etc. No-one else does this ... But this is only one thing Amazon is doing wrong ...

Originally Posted by jlinear

Funny how Amazon deletes my "diddled" Amazon purchased Calibre sideloaded content when I have absolutely no such problems using my "diddled" Amazon purchased Calibre sideloaded content on my Kobo devices.

I think it's great that some people are happier with Kobo. I think everyone who objects to how Amazon has designed their ecosystem and is dissatisfied with the workarounds should promptly go out and get a Kobo (or other) device. No one's stopping you!

And if you already use a non-Kindle device, why are complaining about Amazon/Kindle in the Kindle forum? Are you hoping to get other Kindle users to join you? Are you hoping Amazon will read your complaints and change their practices? Why not just walk away from Amazon/Kindle and be done with it?

[Sirtel]

Originally Posted by odamizu

Well, an argument could be made that users shouldn't violate the Kindle Store Terms of Use by stripping DRM, diddling with Kindle ebooks, and sideloading diddled Kindle ebooks with a fake cdetype EBOK tag.

Why do you think they are necessarily Kindle store books? They may be public domain or otherwise DRM-free ebooks converted to AZW3 with Calibre.