Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Readers > Amazon Kindle

Notices

Reply
 
Thread Tools Search this Thread
Old 07-08-2018, 03:57 PM   #1
j.p.s
Wizard
j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.
 
Posts: 1,669
Karma: 27662338
Join Date: Apr 2011
Device: pb360
Easily fix egregious X-ray errors

X-ray files are generated by some automatic method that works pretty good for finding the location of each reference to an entity in the book. The description of an entity is usually retrieved from goodreads (formerly shelfari) with wikipedia as a fallback and the entity's initial mention in the book as a final fallback.

Sometimes the description from wikipedia is for an incorrect entity. Examples include Nichoas Tolstoy in War and Peace and James Blake in Quiet- The Power of Introverts....

Since the XRAY.entities file is an SQLite database in a separate file from the book and the entity_description table is not tied to locations in the book, these errors are easy to fix using only any tool which can manipulate an SQLite database, such as the command line sqlite3 or the GUI sqlitebrowser, both of which run on many platforms.

Example fixes for the above errors follow. They are not comprehensive, I am just selecting a single error from each XRAY file to illustrate how to fix errors.

For War and Peace, the SQL statement to fix the error is:
Code:
UPDATE "entity_description" SET text = "Grand marshal of the Russian court in 1805; Member of the Tsar's suite in 1812.", source = 1 where entity = 15;
The above can be put in a file named XRAY.fix.B00JCDK5ME.sql
For convenience, I symbolically link to it with WaP.fix.sql

and for Quiet...:
Code:
UPDATE "entity_description" SET text="James F. Blake (April 14, 1912 - March 21, 2002) was the bus driver who Rosa Parks defied in 1955, prompting the Montgomery Bus Boycott.", source=1 where entity = 103;
which I name XRAY.fix.B004J4WNL2.sql linked to by Quiet.fix.sql

To fix with sqlite3:
Code:
sqlite3 XRAY.entities.B00JCDK5ME.asc < XRAY.fix.B00JCDK5ME.sql
sqlite3 XRAY.entities.B004J4WNL2.asc < XRAY.fix.B004J4WNL2.sql
If you didn't directly update the XRAY.entities.* files on your Kindle, copy the appropriate XRAY.entities. file to the .sdr directory for the book on your Kindle.

To use sqlitebrowser, use it to open the XRAY.entities. file
click the "Execute SQL" tab
paste the contents of the XRAY.fix...sql file into the "SQL string" window
click the "Execute query" button
click the Save (floppy disk) icon

In the above SQL statements, "1" in the source field means wikipedia, I think NULL for source means the description came from the book itself and that "2" means shelfari (goodreads).

I think it would be good to establish a repository of XRAY.fix...sql files, maybe on the mobileread wiki or github or both. Is there any interest in that?
j.p.s is online now   Reply With Quote
Old 07-21-2018, 09:35 PM   #2
j.p.s
Wizard
j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.
 
Posts: 1,669
Karma: 27662338
Join Date: Apr 2011
Device: pb360
The original meaning of this thread is that it is easy to fix individual X-ray mistaken identity errors yourself, and that still applies. But it can be time consuming and tedious if there are a large number of such errors in a single book. However, once the file to make the fix is constructed and published, it is still quick and easy for others to apply to the same book. The same fix file should work for both KF8 (AZW3) and KFX forms of the book even though the XRAY.entity.{ASIN}.asc files for the two forms are different.

I've created a repository at https://github.com/jps-e/fix_xray which currently has fixes for two books,
Quiet ... by Susan Cain and War and Peace by Leo Tolstoy. Each probably has more errors, which I will fix as I find them or they get reported as an issue at the repository or in my thread https://www.mobileread.com/forums/sh...d.php?t=309190

I am also soliciting reports for other books at the repository and in the above thread.

I've made a fix for the over 300 errors I found in the the XRAY file for SPQR by Mary Beard, added that to the above github repository released as v0.1.2.

The latest release is attached here and all releases are available at the repository in tar.gz and zip format.
Attached Files
File Type: zip fix_xray-0.1.3.zip (15.3 KB, 2 views)

Last edited by j.p.s; 12-09-2018 at 07:33 PM. Reason: Attach latest release
j.p.s is online now   Reply With Quote
Advert
Old 07-23-2018, 06:37 PM   #3
jhowell
Wizard
jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.
 
jhowell's Avatar
 
Posts: 2,454
Karma: 24993067
Join Date: Nov 2011
Location: Florida
Device: Oasis 2, Fire, iPad Air 2, Nexus 7
Amazon released Kindle for iOS version 6.9 today and one of the new features is "A re-designed X-Ray makes exploring the contents of your book easier." I am not all that familiar with X-ray so I can't tell what has actually changed.

There is a possibility that this re-design might involve a change to the X-ray file format. Something to look out for in future e-ink Kindle firmware.
jhowell is online now   Reply With Quote
Old 07-24-2018, 12:23 PM   #4
j.p.s
Wizard
j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.
 
Posts: 1,669
Karma: 27662338
Join Date: Apr 2011
Device: pb360
Quote:
Originally Posted by jhowell View Post
Amazon released Kindle for iOS version 6.9 today and one of the new features is "A re-designed X-Ray makes exploring the contents of your book easier." I am not all that familiar with X-ray so I can't tell what has actually changed.

There is a possibility that this re-design might involve a change to the X-ray file format. Something to look out for in future e-ink Kindle firmware.
Thanks for the warning. I'm hoping that it is either limited to a change in the reading app or a compatible extension to the database. Certainly they have completely redone the way they store the data at least once before. Do you know whether the previous iOS version used a XRAY.entites.{ASIN}.asc file and what sort of X-ray related files are in the new version?
j.p.s is online now   Reply With Quote
Old 07-24-2018, 02:07 PM   #5
jhowell
Wizard
jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.
 
jhowell's Avatar
 
Posts: 2,454
Karma: 24993067
Join Date: Nov 2011
Location: Florida
Device: Oasis 2, Fire, iPad Air 2, Nexus 7
Quote:
Originally Posted by j.p.s View Post
Do you know whether the previous iOS version used a XRAY.entites.{ASIN}.asc file and what sort of X-ray related files are in the new version?
I don't know. None of my iOS devices are jailbroken.
jhowell is online now   Reply With Quote
Advert
Old 11-23-2018, 02:43 PM   #6
j.p.s
Wizard
j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.
 
Posts: 1,669
Karma: 27662338
Join Date: Apr 2011
Device: pb360
I've updated post #2 in this thread to reflect that my fixes for SPQR have been added to the github repository and attached for convenience to past #2 as fix_xray-0.1.2.zip

SPQR is an excellent book and its everyday US price for the Kindle edition is lower than its UK daily deal price earlier this year ($5.57 but, as usual for the US, slightly higher than the hardcover price and substantially higher than the paperback price). Its many illustrations render reasonably well on e-ink and the maps are legible when zoomed. I found the XRAY feature helpful while reading, but in my opinion the fixes are greatly needed. Wouldn't you think the following would be for two very different books?
Code:
Name             Mentions
Caligula         546
Cicero's brother 456
Code:
Name             Mentions
Cicero           410
Caesar Augustus  289
Julius Caesar    243
Cicero's brother  13
Caligula           9
Cicero's son       3
j.p.s is online now   Reply With Quote
Old 11-24-2018, 06:59 AM   #7
FrustratedReader
Writes SF&F
FrustratedReader ought to be getting tired of karma fortunes by now.FrustratedReader ought to be getting tired of karma fortunes by now.FrustratedReader ought to be getting tired of karma fortunes by now.FrustratedReader ought to be getting tired of karma fortunes by now.FrustratedReader ought to be getting tired of karma fortunes by now.FrustratedReader ought to be getting tired of karma fortunes by now.FrustratedReader ought to be getting tired of karma fortunes by now.FrustratedReader ought to be getting tired of karma fortunes by now.FrustratedReader ought to be getting tired of karma fortunes by now.FrustratedReader ought to be getting tired of karma fortunes by now.FrustratedReader ought to be getting tired of karma fortunes by now.
 
FrustratedReader's Avatar
 
Posts: 144
Karma: 2152630
Join Date: Jun 2017
Device: Kobo H20, PW3, Simple Touch, Readme 2282, Sony PRS350, Kindle DXG
Is X-Ray that important for reading a novel at all, or simply an Amazon marketing gimmick?
FrustratedReader is offline   Reply With Quote
Old 12-08-2018, 02:36 PM   #8
j.p.s
Wizard
j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.
 
Posts: 1,669
Karma: 27662338
Join Date: Apr 2011
Device: pb360
Quote:
Originally Posted by FrustratedReader View Post
Is X-Ray that important for reading a novel at all, or simply an Amazon marketing gimmick?
Whether or not X-Ray is important for reading anything is subjective. For what it is worth, the X-ray builder thread has over 100,000 views.

I would think that keeping characters straight in a long Russian novel with lots of characters, many of which referenced by multiple different names, is the quintessential use case for X-ray. I haven't read War and Peace or any other novel with X-ray yet, but I have found it useful for the half dozen non-fiction books that I have read with it.

One disappointment for me is the large number of errors that could be quickly and easily spotted in cursory checks by (reasonably bright) high school interns at the publisher. This thread is about fixing these errors without even needing to remove DRM or modify the book in any way.
j.p.s is online now   Reply With Quote
Old 12-09-2018, 07:39 PM   #9
j.p.s
Wizard
j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.j.p.s ought to be getting tired of karma fortunes by now.
 
Posts: 1,669
Karma: 27662338
Join Date: Apr 2011
Device: pb360
I've finished fixing the X-ray errors I found in The Tangled Tree by David Quammen. The fix has been committed to the repository whose latest release has been attached above to replace the previous attachment.

Summary of fixes:
Spoiler:

Fix description for George Fox.
Fix description for James R Brown.
Fix description for Eric Lyons.
Fix description for Larry Gold.
Fix description for Bernadette Pace.
Fix description for Institute for Genomic Biology.
Fix description for Institute for Genomic Research.
Remove false occurrances of entity 166 (Norman R. Pace) from occurrence.
Move IGB occurrences from TIGR to IGB.
Fix mixups between Norman, Bernadette, and Norman Pace.
Move Bernadette occurrences in John Pace to Bernadette Pace.
Move Norman occurrances in John Pace to Norman Pace.
Remove incorrect references from Alexander Humboldt.
Remove incorrect references to The Institute for Genomic Research.
j.p.s is online now   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Suggestion: Fix all errors of certain type phossler Editor 4 02-08-2014 10:52 PM
EPubCheck errors -- any idea how to fix? eggheadbooks1 ePub 20 07-28-2011 01:43 PM
Can anyone tell me how to fix these validation errors? grapplegirl ePub 9 12-18-2010 10:23 AM


All times are GMT -4. The time now is 07:14 PM.


MobileRead.com is a privately owned, operated and funded community.