Troubleshooting X-ray functionality for "War and Peace"

[DuckieTigger]

Originally Posted by j.p.s

After weeks of fruiltless chat and email exchanges with Amazon, a good X-ray file for War and Peace was delivered to my Kindle. They would never reveal whether they investigated whether something, somewhere, was broken with this specific book. Until a couple of weeks ago, they never made any promised follow ups. When they finally did, it was to apologize and provided a hard deadline.

When that plus 2 days passed, I contacted them again by chat, which was immediately passed to a specialist who insisted that the fix was to upgrade the firmware, even though I've bought several books since opening the issue where X-ray works just fine. I stewed on that for a few days and finally decided to jump through 1 more hoop to humor them. But first I did one more delete and reload of the book and this time an X-ray 250% the size of the original was delivered that worked, with no firmware update performed.

I guess I will never know what was causing the problem.

I do know that this has soured me on trying to interact with amazon customer service for anything other than problems with physical objects.

Ah, thanks for that. I deleted my book and got the new one from the cloud with all of x-ray working. Amazon fixed the book behind our backs -- I have not clicked update or have automatic book updates enabled. Maybe because they did not actually update the book itself as download via usb still gives the same file size (except for missing x-ray functionality).

[j.p.s]

Originally Posted by DuckieTigger

Ah, thanks for that. I deleted my book and got the new one from the cloud with all of x-ray working. Amazon fixed the book behind our backs -- I have not clicked update or have automatic book updates enabled. Maybe because they did not actually update the book itself as download via usb still gives the same file size (except for missing x-ray functionality).

I don't think the problem was caused by the book itself, it was the Xray file or how it got delivered that needed to be fixed. I don't know what triggers the xray download, so I always used the menu to delete the book and retransfer by usb to trigger it as new on the device. I suspect everyone out there with a bad xray file won't get a replacement until they do whatever actually triggers an xray download.

Big thanks to everyone that looked into this and reported their findings.

[DuckieTigger]

Originally Posted by j.p.s

I don't think the problem was caused by the book itself, it was the Xray file or how it got delivered that needed to be fixed. I don't know what triggers the xray download, so I always used the menu to delete the book and retransfer by usb to trigger it as new on the device. I suspect everyone out there with a bad xray file won't get a replacement until they do whatever actually triggers an xray download.

Big thanks to everyone that looked into this and reported their findings.

Did that actually ever work? When I download & transfer via USB with quickstop in calibre for removal of DRM, I never get X-Ray file downloaded.

I don't mind either way - archive book without X-Ray and without DRM is good enough for me. I can read comfortably with KFX or KF8 files, so downloading from my Kindle is never inconvenient. The only problem is my font ramp - on KFX I use size 5, on KF8 I need 9. One of these days I am going to change my ramp to copy the 9 spot to 5, so it is the same either way ...

[PoP]

Originally Posted by DuckieTigger

Did that actually ever work?...

Yes, that is also the way it worked for me.

Originally Posted by DuckieTigger

...When I download & transfer via USB with quickstop in calibre for removal of DRM, I never get X-Ray file downloaded.
...

After the book has been touched by Calibre it no longer is the same book and it won't trigger the X-Ray download from Amazon. Even if it did, the X-Ray entities locations won't match the locations in the de-DRMed book. You can try this by using Amazon's X-Ray files with the sideloaded book, X-Ray seems functional but give irrelevant references.

[DuckieTigger]

Originally Posted by PoP

Yes, that is also the way it worked for me.



After the book has been touched by Calibre it no longer is the same book and it won't trigger the X-Ray download from Amazon. Even if it did, the X-Ray entities locations won't match the locations in the de-DRMed book. You can try this by using Amazon's X-Ray files with the sideloaded book, X-Ray seems functional but give irrelevant references.

Ah, I see. Thanks for that. I will just stick to delivering to or downloading from Kindle, and do a backup when I make it into the computer room.

[j.p.s]

Originally Posted by DuckieTigger

Did that actually ever work? ...

Originally Posted by PoP

Yes, that is also the way it worked for me.

That is what I've been doing.

Originally Posted by PoP

After the book has been touched by Calibre it no longer is the same book and it won't trigger the X-Ray download from Amazon. Even if it did, the X-Ray entities locations won't match the locations in the de-DRMed book. You can try this by using Amazon's X-Ray files with the sideloaded book, X-Ray seems functional but give irrelevant references.

My (limited) experience is a bit different. After Calibre, the file is the same size and the xray file works fine with either. I agree that an xray file from the KFX form of the book will give garbage references for an AZW3 book and vice versa.

You have goaded me into an experiment. Doing an od -a (like hex dump, but ascii like characters) on both gave results with 3 small blocks of differences near the top.

od -a precalbook.azw3 > precalbook.txt
od -a postcalbook.azw3 > postcalbook.txt

The parts that were the same had the same offset from the beginning of the file, so xray matching should be perfect.

[DuckieTigger]

So, possibly removing of DRM is safe, but editing metadata might not be as it changes file size. Either not triggering the download of X-Ray or making it corrupt. Makes sense that DRM alone would not effect locations - otherwise syncing between different devices might not work.

As an afterthought, it is very practical that a new X-Ray file does not need to be generated everytime DRM is applied - what nightmare would that be for Amazon.

[PoP]

Originally Posted by j.p.s

That is what I've been doing.


My (limited) experience is a bit different. After Calibre, the file is the same size and the xray file works fine with either. I agree that an xray file from the KFX form of the book will give garbage references for an AZW3 book and vice versa.

You have goaded me into an experiment. Doing an od -a (like hex dump, but ascii like characters) on both gave results with 3 small blocks of differences near the top.

od -a precalbook.azw3 > precalbook.txt
od -a postcalbook.azw3 > postcalbook.txt

The parts that were the same had the same offset from the beginning of the file, so xray matching should be perfect.

Thanks for pointing the usefulness of od -a .

In addition to removing DRM, by "touching with Calibre", I meant converting AZW3 to AZW3 or KFX to KFX with the KFX input plugin. This indeed does modify the file length and does cause invalid X-Ray references.

Per your analysis I have to agree that X-Ray matching should be perfect when only sideloading

I never was succesful in testing this though, when I replaced amazon "War and Peace" by its de-DRMed sideloaded copy I never saw the book appearing on the Kindle. It obstinately gave me a message saying "1 item pending download from the cloud" until I gave in, deleted everyting -- sdr directory and book -- and re-downloaded a fresh copy from amazon.

Makes me wonder of the correct procedure for moving an original sdr directory + its de-DRMed book to another ereader.

[j.p.s]

Originally Posted by PoP

Thanks for pointing the usefulness of od -a .

You're welcome.

I usually use od -t x1 for this sort of thing, but many flavors of od do not support the -t option.

Originally Posted by PoP

In addition to removing DRM, by "touching with Calibre", I meant converting AZW3 to AZW3 or KFX to KFX with the KFX input plugin. This indeed does modify the file length and does cause invalid X-Ray references.

Per your analysis I have to agree that X-Ray matching should be perfect when only sideloading

I never was succesful in testing this though, when I replaced amazon "War and Peace" by its de-DRMed sideloaded copy I never saw the book appearing on the Kindle. It obstinately gave me a message saying "1 item pending download from the cloud" until I gave in, deleted everyting -- sdr directory and book -- and re-downloaded a fresh copy from amazon.

Makes me wonder of the correct procedure for moving an original sdr directory + its de-DRMed book to another ereader.

Even before getting interested in X-ray, I have only ever copied the azw3 file and when I open it a fresh .sdr directory gets created. I've never regretted not getting benefit from whatever was in the original .sdr; it was a pleasant surprise to find that copying the XRAY.entities.{ASIN}.asc file to the new .sdr works fine.

Managing books seems to be getting more complicated. As near as I can tell, copying an AZW3 directly to /mnt/us/documents then turning airplane mode off, the Kindle will delete the AZW3 and download the KFX version of the book if there is one. But if I do the same except to a subdirectory of documents, the Kindle leaves the AZW3 alone and downloads an XRAY file if there is one for the book.

[PoP]

Originally Posted by j.p.s

Managing books seems to be getting more complicated
...
But if I do the same except to a subdirectory of documents, the Kindle leaves the AZW3 alone and downloads an XRAY file if there is one for the book.

Indeed ...Ahhh, I see.

[j.p.s]

Amazon's XRAY file for the War and Peace version in this thread has two
entities that I think are the same character. I haven't read the book yet and
part of why I want to do so with a reasonably good XRAY file is to keep
characters straight, especially Russians with a confusing array of names. (And
then there all the mind numbing different transliterations used in different
sources.)

Entity | Name                     | Number of mentions
90     | Prince Vasili Sergeevich | 225
92     | Prince Vasili Kuragin    |  17

I can find Vasili Kuragin (a prince) in:
https://en.wikipedia.org/wiki/List_o...ace_characters
but no trace of Vasili Sergeevich.

Can anyone confirm whether these are the same character?

[DuckieTigger]

According to: https://www.macalester.edu/~hammarbe.../Whoiswho.html

It may just be multiple last names and same person:

Kuragin family:
Prince Vasili Sergeevich Kuragin

I am not familiar with naming convention, but looking through that list makes it appear that the one last name "Sergeevich" is in a spot that refers to one's ancestors family.

[Lady Stardust]

Originally Posted by j.p.s

Amazon's XRAY file for the War and Peace version in this thread has two
entities that I think are the same character. I haven't read the book yet and
part of why I want to do so with a reasonably good XRAY file is to keep
characters straight, especially Russians with a confusing array of names. (And
then there all the mind numbing different transliterations used in different
sources.)

Entity | Name                     | Number of mentions
90     | Prince Vasili Sergeevich | 225
92     | Prince Vasili Kuragin    |  17

I can find Vasili Kuragin (a prince) in:
https://en.wikipedia.org/wiki/List_o...ace_characters
but no trace of Vasili Sergeevich.

Can anyone confirm whether these are the same character?

Yes, as Tigger says, it is the same person. Sergeevich is patronymic (his father is Sergey) and family name is Kuragin.

[j.p.s]

Originally Posted by DuckieTigger

According to: https://www.macalester.edu/~hammarbe.../Whoiswho.html

It may just be multiple last names and same person:

I am not familiar with naming convention, but looking through that list makes it appear that the one last name "Sergeevich" is in a spot that refers to one's ancestors family.

Originally Posted by Lady Stardust

Yes, as Tigger says, it is the same person. Sergeevich is patronymic (his father is Sergey) and family name is Kuragin.

Thanks to you both.

This is turning out to be even more complicated than I thought.

I had planned to eliminate entity 92 and its description and change all references of 92 to 90 and change the label for 90 to "Prince Vasili" which would match both. But for entity 92 the first "i" in Vasili isn't an ASCII "i" but an UTF character where the dot is an accent mark. (Please don't flame me on terminology, my cultural written language is 7 bit ASCII.) Also for entity 92, the "a" in Kuragin has an accent mark. It also turns out that there is an entity 708 "Prince Vasili Kuragin" (All 7 bit ASCII).

I don't know whether it is possible to make the XRAY.entities file correct and consistent without changing the text of the book. I guess that I will stick to fixing mistaken identities in XRAY.entities files, there are certainly plenty of them.

[Ephemerality]

Originally Posted by j.p.s

I don't know whether it is possible to make the XRAY.entities file correct and consistent without changing the text of the book. I guess that I will stick to fixing mistaken identities in XRAY.entities files, there are certainly plenty of them.

It doesn't actually do any matching from the X-Ray file unless it's changed recently, so you should be good to use whichever entity you want to keep, change the references to that entity id (in entity_description, entity_exerpt, exerpt, and occurrence).