MobileRead Forums - View Single Post - Content Easily fix egregious X-ray errors

j.p.s · 07-08-2018, 02:57 PM

X-ray files are generated by some automatic method that works pretty good for finding the location of each reference to an entity in the book. The description of an entity is usually retrieved from goodreads (formerly shelfari) with wikipedia as a fallback and the entity's initial mention in the book as a final fallback.

Sometimes the description from wikipedia is for an incorrect entity. Examples include Nichoas Tolstoy in War and Peace and James Blake in Quiet- The Power of Introverts....

Since the XRAY.entities file is an SQLite database in a separate file from the book and the entity_description table is not tied to locations in the book, these errors are easy to fix using only any tool which can manipulate an SQLite database, such as the command line sqlite3 or the GUI sqlitebrowser, both of which run on many platforms.

Example fixes for the above errors follow. They are not comprehensive, I am just selecting a single error from each XRAY file to illustrate how to fix errors.

For War and Peace, the SQL statement to fix the error is:

Code:

UPDATE "entity_description" SET text = "Grand marshal of the Russian court in 1805; Member of the Tsar's suite in 1812.", source = 1 where entity = 15;

The above can be put in a file named XRAY.fix.B00JCDK5ME.sql
For convenience, I symbolically link to it with WaP.fix.sql

and for Quiet...:

Code:

UPDATE "entity_description" SET text="James F. Blake (April 14, 1912 - March 21, 2002) was the bus driver who Rosa Parks defied in 1955, prompting the Montgomery Bus Boycott.", source=1 where entity = 103;

which I name XRAY.fix.B004J4WNL2.sql linked to by Quiet.fix.sql

To fix with sqlite3:

Code:

sqlite3 XRAY.entities.B00JCDK5ME.asc < XRAY.fix.B00JCDK5ME.sql
sqlite3 XRAY.entities.B004J4WNL2.asc < XRAY.fix.B004J4WNL2.sql

If you didn't directly update the XRAY.entities.* files on your Kindle, copy the appropriate XRAY.entities. file to the .sdr directory for the book on your Kindle.

To use sqlitebrowser, use it to open the XRAY.entities. file
click the "Execute SQL" tab
paste the contents of the XRAY.fix...sql file into the "SQL string" window
click the "Execute query" button
click the Save (floppy disk) icon

In the above SQL statements, "1" in the source field means wikipedia, I think NULL for source means the description came from the book itself and that "2" means shelfari (goodreads).

I think it would be good to establish a repository of XRAY.fix...sql files, maybe on the mobileread wiki or github or both. Is there any interest in that?

07-08-2018, 02:57 PM	#1
j.p.s Grand Sorcerer Posts: 5,808 Karma: 103362673 Join Date: Apr 2011 Device: pb360	Easily fix egregious X-ray errors X-ray files are generated by some automatic method that works pretty good for finding the location of each reference to an entity in the book. The description of an entity is usually retrieved from goodreads (formerly shelfari) with wikipedia as a fallback and the entity's initial mention in the book as a final fallback. Sometimes the description from wikipedia is for an incorrect entity. Examples include Nichoas Tolstoy in War and Peace and James Blake in Quiet- The Power of Introverts.... Since the XRAY.entities file is an SQLite database in a separate file from the book and the entity_description table is not tied to locations in the book, these errors are easy to fix using only any tool which can manipulate an SQLite database, such as the command line sqlite3 or the GUI sqlitebrowser, both of which run on many platforms. Example fixes for the above errors follow. They are not comprehensive, I am just selecting a single error from each XRAY file to illustrate how to fix errors. For War and Peace, the SQL statement to fix the error is: Code: UPDATE "entity_description" SET text = "Grand marshal of the Russian court in 1805; Member of the Tsar's suite in 1812.", source = 1 where entity = 15; The above can be put in a file named XRAY.fix.B00JCDK5ME.sql For convenience, I symbolically link to it with WaP.fix.sql and for Quiet...: Code: UPDATE "entity_description" SET text="James F. Blake (April 14, 1912 - March 21, 2002) was the bus driver who Rosa Parks defied in 1955, prompting the Montgomery Bus Boycott.", source=1 where entity = 103; which I name XRAY.fix.B004J4WNL2.sql linked to by Quiet.fix.sql To fix with sqlite3: Code: sqlite3 XRAY.entities.B00JCDK5ME.asc < XRAY.fix.B00JCDK5ME.sql sqlite3 XRAY.entities.B004J4WNL2.asc < XRAY.fix.B004J4WNL2.sql If you didn't directly update the XRAY.entities.* files on your Kindle, copy the appropriate XRAY.entities. file to the .sdr directory for the book on your Kindle. To use sqlitebrowser, use it to open the XRAY.entities. file click the "Execute SQL" tab paste the contents of the XRAY.fix...sql file into the "SQL string" window click the "Execute query" button click the Save (floppy disk) icon In the above SQL statements, "1" in the source field means wikipedia, I think NULL for source means the description came from the book itself and that "2" means shelfari (goodreads). I think it would be good to establish a repository of XRAY.fix...sql files, maybe on the mobileread wiki or github or both. Is there any interest in that?