X-Ray Builder - Windows Application

[Ephemerality]

For those who don't already know, X-Ray is a reference feature available on any Kindle product with a touch screen (Touch, Paperwhite, Kindle for Android, etc), provided the book has an X-Ray available for it.

X-Ray Builder gives users the ability to create fully-functional X-Rays for both retail books that Amazon has not X-Ray-enabled yet as well as side-loaded books that have been purchased elsewhere.

It will also build some extra metadata files:

• Author Profile - Enables "About the author" button
• Start Actions - "About this book" info which usually has a description of the book, author, and some recommendations
• End Actions - Pops up when you're done reading the book to provide recommendations on what to read next

Goodreads is used as the main datasource, but does not always contain rich character information. A new source called Roentgen is now available that may have more data available depending on the book.

Requirements:

• Windows 7 or higher (but really you should be on 10 by now)
• .Net Framework 4.6.2
• DRM-free books to process!
• You may need to install the Microsoft Visual C++ 2010 x86 Redistributable

Instructions:
See camelx's step-by-step guide and the Quality Check plugin.
You can also check out the basic video demo that gives you a 5min rundown of how to use the program.

May 2020 Note
These instructions are outdated and may not be entirely accurate anymore.
I'll try to update them at some point...

In general you should be able to:

• Open the book (ideally fix the ASIN if it prompts & successfully finds the book on Amazon)
• Click the search button to get the Goodreads link
• Click the checkmark "1-click" button to start building all files
• Answer the prompts during the build process
• Either everything worked or the log will show what went wrong (depending on how many terms were found you may want to switch sources)

Terms Builder (useful for users creating translated X-Rays)
There is a button for downloading all terms to an XML file as well as a simple interface for adding/modifying them.
When available, terms can also be imported into the terms builder directly from Roentgen.

Creating the perfect X-Ray
Creating a perfect X-Ray is not a 1-step process, but a decent one is usually just a few clicks away.
The default options will usually give you a reasonable result but getting the best results may require a bit of effort.
Character aliases usually need to be used to match more locations within the book, for example if the character name is "Harry Potter", the builder won't know that you also want to match the word "Harry" as well.
For some books, a last name might be exclusive to a character and used on its own - making "Potter" a valid alias. In most cases this isn't possible, as it would also match any other spot where the last name shows up - "James Potter."
This requires knowing how the names are used throughout the book, which mostly defeats the purpose of having the X-Ray in the first place.

Alias Files:

Spoiler:

Special options:

• Automatically split aliases - This will match first and last names as well as common titles like "Captain" and "Mr." Use this feature carefully as it can cause incorrect matches in some cases.

After downloading the terms, they will be exported to a .aliases file in the ext folder, named after the book's ASIN.
Characters do not usually show up by their full name throughout the text. The alias file allows you to define aliases for characters/topics manually to maximize the number of excerpts found within the book.

Aliases follow the format:
Character Name|Alias1,Alias2,Etc
John Smith|Mr. Smith,Johnny,John

Ensure that any aliases that are very basic, eg John, are at the end. Otherwise, if you have a setup like "John Smith|John,Johnny", "John" will always match before "Johnny" does and it will look weird when you're viewing the X-Ray.

X-Ray Builder will tell you if no excerpts were found when processing. If you have never read the book before, you can ignore this if you wish; the X-Ray will not be ideal but it will work. For characters with a first and last name, you can usually assume that their first name will be used and can add an alias for it (John Smith|John).

There are a number of switches that can be used to change the matching behavior for that character only:

• /c - Searches will be case-sensitive
• /d - Delete this character (not all terms on Shelfari may be desired)
• /n - Will not match any excerpts but will leave character in X-Ray
• /r - Aliases will be processed as regular expressions (case-sensitive unless specified). ONLY the aliases will be used in this case; the character's original name (to the left of the | symbol) WILL NOT BE USED.

Only one switch can be used per character and should be used like this:
Character Name|/c,Alias1,Alias2

For more information on regular expressions for .NET check out this page.
For creating and testing expressions, RegExr is a great resource.

Troubleshooting

Spoiler:

X-Ray is overwritten when not using airplane mode
If your book has X-Ray enabled by Amazon, your Kindle may try to download theirs.
If you want to keep your own, you will need to set the file to read-only when you copy it to the Kindle.

X-Ray button does not show up:

• File was not placed in the correct location. Should be in the .sdr folder for your book.
• If you converted your book using Calibre, ensure the converted copy of the book is copied back to your Kindle (from the Calibre Library).
• Book is tagged as a personal document. Copy your book from the Calibre Library *after* building the X-Ray.

X-Ray button shows up but does not open and the X-Ray file you copied is deleted:

• Ensure the "use new format" option is enabled in the settings.

Acknowledgements
Thanks to:

• jhowell for the KFX Input/Output plugins
• darrenmcg for completely overhauling the interface and adding many new features
• shinew for the original X-Ray creation site and source
• Everyone who has used the program so far, especially those who have reported any errors or suggested new features

Source:
GUI Version: Ephemerality/xray-builder.gui

Error Reporting:
Any issues with the program can be posted here (if you paste any logs, please put them in a spoiler tag), PMed directly to me, or posted as an issue on GitHub.
A copy of the book can be very helpful as well and can be sent to me if you want. Books are used for testing purposes only and deleted afterwards (must be DRM-free).

Kindle app for Android
There is now an option in Settings to turn on building for Android. These files will be placed in the "out\Android\BXXXXXXXXX" folder, where the name is the book's ASIN.
Your book needs to go in the "sdcard/Android/data/com.amazon.kindle/files/" folder on your Android device.
You can then copy the entire BXXXXXXXXX folder to that same spot.

Calibre Plugin
This plugin (plugin-kindlexray-0.0.3.zip) allows Calibre to automatically find and copy the files built by X-Ray Builder to your Kindle when you use the 'send to device' feature.
To install, load the plugin from the zip file from Preferences -> Plugins as per usual.
Once installed, it will show up under 'Device Interface plugins'.
You should disable the original 'Kindle 2/3/4/Touch/etc' plugin to ensure it uses the new one.
Select the KindleXRay plugin and hit 'Customize plugin'. You will need to hit browse and point it to your X-Ray Builder's /out folder so it knows where to look. You can choose whether or not to overwrite existing X-Ray/extras files as well.
The plugin gives no feedback on success, but if all goes well, when you send a book to your Kindle, the files you created in X-Ray Builder will copy over as well.
If not, you can try starting Calibre in debug mode, copy the book to your device, then close Calibre to view the log to check for any exceptions.

Note regarding Kindle firmware < 5.6
If you have a Kindle Paperwhite with firmware version lower than v5.6, you will have to uncheck "Use New X-Ray Format" in the settings page.
The following section is retained for legacy purposes in case anyone actually needs it.

Spoiler:

Chapters:
In many cases, chapters are automatically detected. If chapters were found, they are exported to a .chapters file in the ext folder.
You will be prompted to edit the chapters in Notepad. This allows you to remove any random chapters you may not want included, like copyright pages, acknowledgements, etc. You can also set up 'parts' in case the book is divided into parts/sections that include multiple chapters.

If chapters are not detected, you can build them yourself. You need to open the raw markup and use a text editor that allows you to see the file locations, such as Notepad++.
Chapter format:
Name|start|end

Troubleshooting:

X-Ray file does not load, shows a 2-character code at the bottom of the page:

• SX = X-Ray file not valid
• VS/VN/VO = Version number is anything but "1"
• WA = Issue with ASIN
• WG = Issue with GUID (make sure it isn't too long, X-Ray builder will usually warn you)

[pakiyabhai]

Hello,

Whenever I use the GUI program,at the end when the scanning book content start , I get an error saying

Spoiler:

Unhandled exception has occurred in your application.


This is the detailed log
See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.

************** Exception Text **************
System.NullReferenceException: Object reference not set to an instance of an object.
at XRayBuilderGUI.XRay.expandFromRawML(String rawML)
at XRayBuilderGUI.frmMain.btnBuild_Click(Object sender, EventArgs e)
at System.Windows.Forms.Control.OnClick(EventArgs e)
at System.Windows.Forms.Button.OnClick(EventArgs e)
at System.Windows.Forms.Button.OnMouseUp(MouseEventAr gs mevent)
at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ButtonBase.WndProc(Message& m)
at System.Windows.Forms.Button.WndProc(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.O nMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.W ndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)


************** Loaded Assemblies **************
mscorlib
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.34209 built by: FX452RTMGDR
CodeBase: file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/mscorlib.dll
----------------------------------------
XRayBuilderGUI
Assembly Version: 1.3.1.0
Win32 Version: 1.3.1.0
CodeBase: file:///D:/Kindle/XRayBuilderGUI.exe
----------------------------------------
System.Windows.Forms
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.34209 built by: FX452RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Windows.Forms/v4.0_4.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
----------------------------------------
System.Drawing
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.34209 built by: FX452RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Drawing/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
----------------------------------------
System
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.34209 built by: FX452RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System/v4.0_4.0.0.0__b77a5c561934e089/System.dll
----------------------------------------
System.Configuration
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.34209 built by: FX452RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Configuration/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
----------------------------------------
System.Core
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.34209 built by: FX452RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Core/v4.0_4.0.0.0__b77a5c561934e089/System.Core.dll
----------------------------------------
System.Xml
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.34209 built by: FX452RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Xml/v4.0_4.0.0.0__b77a5c561934e089/System.Xml.dll
----------------------------------------
HtmlAgilityPack
Assembly Version: 1.4.6.0
Win32 Version: 1.4.6.0
CodeBase: file:///D:/Kindle/HtmlAgilityPack.DLL
----------------------------------------

************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.

For example:

<configuration>
<system.windows.forms jitDebugging="true" />
</configuration>

When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.

[Ephemerality]

Originally Posted by pakiyabhai

Hello,

Whenever I use the GUI program,at the end when the scanning book content start , I get an error saying

Unhandled exception has occurred in your application.

I'll have to add some better error reporting to it to make it a bit easier to solve these. What book was it? If possible, you can send me a copy of the book and I can see if it works on my end. (I delete books people send once I'm done testing them in the program)

[vanbarel]

Hi Ephemerality, the tool is very nice, once I noticed that MOBI files work better (the AZW3 files created by Calibre don't work, there is an issue with Chapter detection and I get the WG error on the kindle)

Just a note: if by chance the chapters file is not created properly but you say you want to open it anyway, the app crashes when you close the empty notepad

[Ephemerality]

Originally Posted by vanbarel

Hi Ephemerality, the tool is very nice, once I noticed that MOBI files work better (the AZW3 files created by Calibre don't work, there is an issue with Chapter detection and I get the WG error on the kindle)

Just a note: if by chance the chapters file is not created properly but you say you want to open it anyway, the app crashes when you close the empty notepad

AZW3s are supposed to work! Would you mind sending me one that you've had issues with and I can take a look at it? I've had a few that have been through Calibre, but they worked alright.

GUI v1.32 has been uploaded to the first post. It should fix the notepad crash and adds some error reporting.

[EbokJunkie]

2Ephemerality
Problem: ASIN containing blank(s) or period gets truncated at these characters.
My books are converted with ASIN equal to file name, that's why I ran into this.
Is this fixable?

[Ephemerality]

Originally Posted by EbokJunkie

2Ephemerality
Problem: ASIN containing blank(s) or period gets truncated at these characters.
My books are converted with ASIN equal to file name, that's why I ran into this.
Is this fixable?

I hadn't seen anyone using a value that wasn't an Amazon ASIN or the generic one that Calibre creates, so it's not matching properly for anything else. I'll upload 1.33 with a fix for it when I get home from work in a couple hours.

[EbokJunkie]

Thanks a lot!
Unfortunately, this is not the last of my woes
During conversion, I'm inserting soft hyphen code (Unicode U+00AD) into all words to provide hyphenation. This prevents your code from recognizing any alias. It would be perfect if you coud add ignoring soft hyphens at the scanning step.

[Ephemerality]

Originally Posted by EbokJunkie

Thanks a lot!
Unfortunately, this is not the last of my woes
During conversion, I'm inserting soft hyphen code (Unicode U+00AD) into all words to provide hyphenation. This prevents your code from recognizing any alias. It would be perfect if you coud add ignoring soft hyphens at the scanning step.

*edited* I think I get what you mean, like hyphenation on line breaks?
A mobi with this in practice would be useful to test on before I release it.

[EbokJunkie]

Ephemerality

http://www.shelfari.com/books/24914/White-Fang

[Ephemerality]

It isn't going to be as easy as I thought it would be so I'm going to put 1.33 up with the ASIN fix and work on the hypen thing. You can delete your attachment now if you want.

[EbokJunkie]

Thank you for your care. Those who use Calibre plugin Hyphenate This! will be grateful.

[Ephemerality]

Originally Posted by EbokJunkie

Thank you for your care. Those who use Calibre plugin Hyphenate This! will be grateful.

I aim to please! However, I haven't found a working solution yet. I have some rough ideas but it will take a while to get anything functional. If I do manage to get it working, I'll send you a copy first to test.
In the meantime I did try adding aliases for terms using soft hypens and it seems to work, but it involves knowing where they are (by searching the rawml).
Such as: Northland|North-land

[vanbarel]

Originally Posted by Ephemerality

AZW3s are supposed to work! Would you mind sending me one that you've had issues with and I can take a look at it? I've had a few that have been through Calibre, but they worked alright.

GUI v1.32 has been uploaded to the first post. It should fix the notepad crash and adds some error reporting.

Actually I will be first looking at the settings in my calibre for MOBI->AZW3 conversion: I probably made some faults there.

I do not have any orginal AZW3, so it's probably a problem in the conversion I make

[gers1978]

I get the following error in Windows 7:



I tried to install .NET 4.0 but Windows said I already had a higher version.