Content Easily fix egregious X-ray errors

[j.p.s]

X-ray files are generated by some automatic method that works pretty good for finding the location of each reference to an entity in the book. The description of an entity is usually retrieved from goodreads (formerly shelfari) with wikipedia as a fallback and the entity's initial mention in the book as a final fallback.

Sometimes the description from wikipedia is for an incorrect entity. Examples include Nichoas Tolstoy in War and Peace and James Blake in Quiet- The Power of Introverts....

Since the XRAY.entities file is an SQLite database in a separate file from the book and the entity_description table is not tied to locations in the book, these errors are easy to fix using only any tool which can manipulate an SQLite database, such as the command line sqlite3 or the GUI sqlitebrowser, both of which run on many platforms.

Example fixes for the above errors follow. They are not comprehensive, I am just selecting a single error from each XRAY file to illustrate how to fix errors.

For War and Peace, the SQL statement to fix the error is:

UPDATE "entity_description" SET text = "Grand marshal of the Russian court in 1805; Member of the Tsar's suite in 1812.", source = 1 where entity = 15;

The above can be put in a file named XRAY.fix.B00JCDK5ME.sql
For convenience, I symbolically link to it with WaP.fix.sql

and for Quiet...:

UPDATE "entity_description" SET text="James F. Blake (April 14, 1912 - March 21, 2002) was the bus driver who Rosa Parks defied in 1955, prompting the Montgomery Bus Boycott.", source=1 where entity = 103;

which I name XRAY.fix.B004J4WNL2.sql linked to by Quiet.fix.sql

To fix with sqlite3:

sqlite3 XRAY.entities.B00JCDK5ME.asc < XRAY.fix.B00JCDK5ME.sql
sqlite3 XRAY.entities.B004J4WNL2.asc < XRAY.fix.B004J4WNL2.sql

If you didn't directly update the XRAY.entities.* files on your Kindle, copy the appropriate XRAY.entities. file to the .sdr directory for the book on your Kindle.

To use sqlitebrowser, use it to open the XRAY.entities. file
click the "Execute SQL" tab
paste the contents of the XRAY.fix...sql file into the "SQL string" window
click the "Execute query" button
click the Save (floppy disk) icon

In the above SQL statements, "1" in the source field means wikipedia, I think NULL for source means the description came from the book itself and that "2" means shelfari (goodreads).

I think it would be good to establish a repository of XRAY.fix...sql files, maybe on the mobileread wiki or github or both. Is there any interest in that?

[j.p.s]

The original meaning of this thread is that it is easy to fix individual X-ray mistaken identity errors yourself, and that still applies. But it can be time consuming and tedious if there are a large number of such errors in a single book. However, once the file to make the fix is constructed and published, it is still quick and easy for others to apply to the same book. The same fix file should work for both KF8 (AZW3) and KFX forms of the book even though the XRAY.entity.{ASIN}.asc files for the two forms are different.

I've created a repository at https://github.com/jps-e/fix_xray which currently has fixes for two books,
Quiet ... by Susan Cain and War and Peace by Leo Tolstoy. Each probably has more errors, which I will fix as I find them or they get reported as an issue at the repository or in my thread https://www.mobileread.com/forums/sh...d.php?t=309190

I am also soliciting reports for other books at the repository and in the above thread.

I've made a fix for the over 300 errors I found in the the XRAY file for SPQR by Mary Beard, added that to the above github repository released as v0.1.2.

The latest release is attached here and all releases are available at the repository in tar.gz and zip format.

[jhowell]

Amazon released Kindle for iOS version 6.9 today and one of the new features is "A re-designed X-Ray makes exploring the contents of your book easier." I am not all that familiar with X-ray so I can't tell what has actually changed.

There is a possibility that this re-design might involve a change to the X-ray file format. Something to look out for in future e-ink Kindle firmware.

[j.p.s]

Originally Posted by jhowell

Amazon released Kindle for iOS version 6.9 today and one of the new features is "A re-designed X-Ray makes exploring the contents of your book easier." I am not all that familiar with X-ray so I can't tell what has actually changed.

There is a possibility that this re-design might involve a change to the X-ray file format. Something to look out for in future e-ink Kindle firmware.

Thanks for the warning. I'm hoping that it is either limited to a change in the reading app or a compatible extension to the database. Certainly they have completely redone the way they store the data at least once before. Do you know whether the previous iOS version used a XRAY.entites.{ASIN}.asc file and what sort of X-ray related files are in the new version?

[jhowell]

Originally Posted by j.p.s

Do you know whether the previous iOS version used a XRAY.entites.{ASIN}.asc file and what sort of X-ray related files are in the new version?

I don't know. None of my iOS devices are jailbroken.

[j.p.s]

I've updated post #2 in this thread to reflect that my fixes for SPQR have been added to the github repository and attached for convenience to past #2 as fix_xray-0.1.2.zip

SPQR is an excellent book and its everyday US price for the Kindle edition is lower than its UK daily deal price earlier this year ($5.57 but, as usual for the US, slightly higher than the hardcover price and substantially higher than the paperback price). Its many illustrations render reasonably well on e-ink and the maps are legible when zoomed. I found the XRAY feature helpful while reading, but in my opinion the fixes are greatly needed. Wouldn't you think the following would be for two very different books?

Name             Mentions
Caligula         546
Cicero's brother 456

Name             Mentions
Cicero           410
Caesar Augustus  289
Julius Caesar    243
Cicero's brother  13
Caligula           9
Cicero's son       3

[Quoth]

Is X-Ray that important for reading a novel at all, or simply an Amazon marketing gimmick?

[j.p.s]

Originally Posted by FrustratedReader

Is X-Ray that important for reading a novel at all, or simply an Amazon marketing gimmick?

Whether or not X-Ray is important for reading anything is subjective. For what it is worth, the X-ray builder thread has over 100,000 views.

I would think that keeping characters straight in a long Russian novel with lots of characters, many of which referenced by multiple different names, is the quintessential use case for X-ray. I haven't read War and Peace or any other novel with X-ray yet, but I have found it useful for the half dozen non-fiction books that I have read with it.

One disappointment for me is the large number of errors that could be quickly and easily spotted in cursory checks by (reasonably bright) high school interns at the publisher. This thread is about fixing these errors without even needing to remove DRM or modify the book in any way.

[j.p.s]

I've finished fixing the X-ray errors I found in The Tangled Tree by David Quammen. The fix has been committed to the repository whose latest release has been attached above to replace the previous attachment.

Summary of fixes:

Spoiler:

Fix description for George Fox.
Fix description for James R Brown.
Fix description for Eric Lyons.
Fix description for Larry Gold.
Fix description for Bernadette Pace.
Fix description for Institute for Genomic Biology.
Fix description for Institute for Genomic Research.
Remove false occurrances of entity 166 (Norman R. Pace) from occurrence.
Move IGB occurrences from TIGR to IGB.
Fix mixups between Norman, Bernadette, and Norman Pace.
Move Bernadette occurrences in John Pace to Bernadette Pace.
Move Norman occurrances in John Pace to Norman Pace.
Remove incorrect references from Alexander Humboldt.
Remove incorrect references to The Institute for Genomic Research.

[j.p.s]

I've finished fixing the X-ray errors I found in Digital Fortress by Dan Brown. The fix has been committed to the repository whose latest release, v0.1.4, has been attached to post #2 above to replace the previous attachment. The fix ot the XRAY file for Digital Fortress took 43 seconds on my laptop because of the large number of additions for entity 42 occurrences. The fixes for the previously fixed books were much faster.

Summary of fixes:

Spoiler:

B003JH8LPW

The description for entity 35, a character referred to as "North Dakota"
was for the US State North Dakota.

Entity 59, "Susan" is the same character as entity 56.

Entity 42, "Trevor Strathmore" was referenced 292 times in Xray, but 668
times in the book,

[j.p.s]

I've finished fixing the X-ray errors I found in The Martian by Andy Weir (found another one during the fix). The fix has been committed to the repository whose latest release, v0.1.5 has been attached to post #2 above. Since it is a full release, it has the fixes for all the books fixed so far and is still less than 20 KB total.

Summary of fixes:

Spoiler:

Fixes #6 The Martian by Andy Weir

B00EMXBDMA

The occurrences of entity 103, "Arabia" (Saudi Arabia), are really occurrences of entity 65, "Arabia Terra", a region in the north of the planet Mars.

The description for entity 16, "Hermes", the spacecraft used in the novel to shuttle Mars mission crews between Earth orbit and Mars orbit, is for the canceled European Space Agency spaceplane "Hermes".

Entity 60, the Martian crater Marth, is mis-identified as the lunar crater Marth.

[j.p.s]

I've finished fixing the X-ray errors I found in The 2020 Commission Report on the North Korean Nuclear Attacks Against the United States by Jeffrey Lewis

The fix has been committed to the repository whose latest release, v0.1.6 has been attached to post #2 above. Since it is a full release, it has the fixes for all the books fixed so far and is slightly more than 20 KB total.

Summary of fixes:

Spoiler:

B079VDR6HM

A number of South Koreans named Park are lumped into xray entity 200
along with the Publisher's address on Park Avenue in NYC on the copyright
page for the book. The entity description is for Park Chung Hee, who is
mentioned once in the book.

Some of the occurrences of entity 94, Major General Lee Jin-won are
actually other characters named Lee.

The character John Sullivan in the book is the acting US secretary of
State, while the xray description of entity 100 is for a John Sullivan
who lived 1740 - 1795.

The xray description for Trump Tower in NYC is for a cancelled project
in Philadelphia.

Entity 68, Ja Song Taek, has only 3 xray references, but is mentioned 42
times in the book.

[j.p.s]

I've added a partial fix for x-ray for The Immortal Life of Henrietta Lacks as an attachment to post #2 in this thread. There are 9 corrections to the descriptions of people where they were completely misidentified. I also changed all occurrences of entity 128 to entity 158 and corrected the count for entity 150.

I might fix the many other errors I found at some later date.

[j.p.s]

The fixes to X-ray for Ready Player One ASIN B004J4WKUQ are attached to post #2 as XrayFixReadyPlayer1.tar.gz

The errors fixed are


The entity description for id 234, MMO, Massively Multiplayer Online game, is for the MMO Report podcast.

The entity description for id 127, Crystal Key, an object in the novel, is for a graphic adventure video game from the year 2000.