X-Ray Builder - Windows Application

[wapkaplet]

Hi all, quick question: I'm toying around with this for my Song of Ice and Fire and Wheel of Time books, as I'm half-way through both series and I'd rather not re-buy them through Amazon. However, it seems like in order to build a proper alias file for either series I'll have to wade through a fair amount of spoilers.

Any chance there's an online resource for alias files, or perhaps someone has some for these series that they'd be willing to share?

Hope I'm asking this in the right place. If not please let me know.

Thanks in advance!

[Ephemerality]

That's really the biggest issue with trying to set up a proper xray for a book you haven't read yet.
Someone may have one done up for those series already, but there is no online resource yet. I have been playing with the idea of setting up an alias database but haven't figured out how it should work yet.
I may have one for ice and fire, I'll have a look when I get home.

Edit: Don't seem to have one saved anymore. I'll make one when I get a chance but it might not be for a few days.

[camelx]

I've messed with this off and on for a couple of months now, but could never get it to work consistent for me. So I spend a few hours on it yesterday and wanted to let the rest of you inconsistent results on how I got it to work for me, did at least 10 books with testing. Mobi and AZW3 and doesn't matter if they are put in the root of Kindle Documents folder or under author like Calibre moves them. I have the newest Voyager and the latest software.

This is the way I got it to work every time, but there may be things that aren't required, I just didn't have more time to prove it all out.

Procedure includes Calibre, the Quality check plug, and lookup metadata for Amazon.

Calibre output settings (preferences): mobi - personal doc tag: [EBOK} and new for file type. (keeps the new books going to kindle NOT as personal docs and brings nice covers

for XRAY config: checked: output UTF8, enable editing of chaps, use new xray format, use subdirs, overwrite existing files. Excerpt offset: 0

For the book you want to Xray:

1) double click on the book in Calibre and delete the existing Ids
2)turn on the Amazon metadata lookup for the sources (I only picked Amazon here, but I'm sure that doesn't matter, but you MUST have Amazon)
3) do a metadata lookup for Amazon and let it write the new metadata
3) Using the Quality Check plugin, choose Fix ASIN for Kindle Fire (don't worry if yours is not a Fire, I have a Voyager!) btw, I found even with Amazon lookup, Xray would NOT always work unless I did this procedure!!!
4) using Calibre, right click on the book and choose Save to disk - you will use either the mobi or the AZW3 from this.
5) run XRay builder and drag one of those files to the book input box
6) use browser and go to www.shelfari.com and look up your book page, once there grab the URL from browser and put it into the URL field of XRay
7) click the build Xray button at top and you should be fine
8) plug your kindle in with USB cable and go to documents directory
9) grab the specific book you used in Xray and the Xray output directory created in the process (in the out directory) and copy both of them directly to the Kindle's documents directory. They should be exactly the same name, but book with extension of mobi or AZW3 and directory ext of .sdr.
10) unplug the usb cable and your book will show up in the kindle and the Xray will be working.

Notes:
It's as only good as the Shelfari information.
XRay is reached by clicking the top part of menu area, not by highlighting a word.
I generally DON"T have Amazon turned on for metadata - not good for tags and book series, I generally only use Goodreads. So after I do the Amazon thing for this, I will go back and only chose Goodreads to fix my metadata the way I like it.

[Ephemerality]

Originally Posted by camelx

I've messed with this off and on for a couple of months now...

Good write up. That basically describes the process I normally go through, with a few variations:

• I don't worry at all about using Amazon metadata within Calibre (I didn't even know about the quality check plugin), I do a normal lookup and usually just leave it at that
• After converting the book in Calibre (I usually use the "both" option rather than new, I forget why) I then open it in Java Mobi Metadata Editor or Mobi2Mobi and edit the EXTH 113/504 fields to the proper ASIN (not sure if 504 is even necessary)
• The new version of the builder has a "Search Shelfari" option so I usually don't have to go find it manually
• If your aliases are set correctly, opening the X-Ray panel should work when highlighting words

Sometime soon I plan on doing a set of video tutorials on the whole process, which will hopefully help.

[eschwartz]

@Ephemerality,

Quality Check is much, much faster for inserting the right exth 113 values.

Don't use "both" -- simply convert to AZW3.
"both" gets you nothing except an ebook that is backwards-compatible with the Kindle DX/2/Original (they don't understand AZW3).
And it comes at a cost --calibre doesn't know how to update the metadata in a dual-MOBI.

[Ephemerality]

Originally Posted by eschwartz

@Ephemerality,

Quality Check is much, much faster for inserting the right exth 113 values.

Don't use "both" -- simply convert to AZW3.
"both" gets you nothing except an ebook that is backwards-compatible with the Kindle DX/2/Original (they don't understand AZW3).
And it comes at a cost --calibre doesn't know how to update the metadata in a dual-MOBI.

Ah, gotcha. I played around with the quality check feature and it seemed quick and easy as long as the "mobi-asin" ID was present in the metadata. I think I still prefer checking the headers manually :P

[Ephemerality]

v2.0.8 has been posted. Notable features:

• Builds a StartActions file when 'extras' are built (StartActions displayed when you hit the "about this book" menu option) - thanks to darrenmcg
• Improved chapter detection - should be able to detect the table of contents in most books that have had one generated and forced in by Calibre (previously could only detect in 'old' format mobis)
• Updated preview windows for X-Ray, End Actions, Start Actions, and Author Profile (darrenmcg)

Since the 'build extras' option needs to pull several pages from the internet, it can freeze the program for a while. Sometime soon, multi-threading support will be added so this does not occur.

[stoduk]

Hi Folks,

I'm new to the land of Kindles, so looking at all the various tools out there. I'm struggling to get the X-Ray generation working with the windows tool (problems with the sqlite DLL, which I've tried and so far failed to fix).

Before I waste time trying to get this working I thought I'd compare how it works with the Calibre plugin. The Calbre plugin is likely to be more convenient for me, but I get the impression that the windows tool is further developed..

To that end, could someone generate the X-ray file for the *free and out-of-copyright* book below? I can't directly link to the mobi as the URLs have a session ID, but just select the "Kindle (with images)" download. I can then compare that with the output from the Calbire plugin and see how thinks fare.

Thanks for helping!
Anthony

http://www.gutenberg.org/ebooks/24022 [A Christmas Carol by Charles Dickens]

[Ephemerality]

Originally Posted by stoduk

Hi Folks,

I'm new to the land of Kindles, so looking at all the various tools out there. I'm struggling to get the X-Ray generation working with the windows tool (problems with the sqlite DLL, which I've tried and so far failed to fix).

To that end, could someone generate the X-ray file for the *free and out-of-copyright* book below? I can't directly link to the mobi as the URLs have a session ID, but just select the "Kindle (with images)" download. I can then compare that with the output from the Calbire plugin and see how thinks fare.

Hey.
What kind of Kindle / firmware version do you have? I'm assuming you need one in the new format?
I had to edit in the ASIN/book type (EBOK) into the mobi file, which metadata/converting/quality check in Calibre would usually take care of. I chose this one, which I think is right (it's also 75 cents on the store ).
What issues are you having with the sqlite dll?
The Calibre plugin does a decent job, but I don't think it has been updated in a while; I don't use it so I don't check up on mj's progress. However, the newest version of this app gives you more features (notable clips, end actions, start actions, etc).

Edit: Attached is a new-format X-Ray along with the End Actions, Start Actions, and Author Profile. I included the aliases/chapter files that I used for building as well.
I included the copy of the book I used; for some reason that I can't explain, the X-Ray button wouldn't show up for me on the version I downloaded, despite having added the proper metadata. This one has been run through Calibre's mobi->mobi conversion with (hopefully) minimal visual difference. The X-Ray won't work properly unless you use this copy with it.
Note: This book is freely distributable in the US under the Project Gutenberg License.

[stoduk]

Hi,

Thanks a lot - very useful!

Originally Posted by Ephemerality

Hey.
What kind of Kindle / firmware version do you have? I'm assuming you need one in the new format?

Sorry, yes it is a Kindle Paperwhite v2 (so a 6th gen kindle), so new format. Not sure what the cut off is for old vs. new format (I imagine it is software version as much as device, but I've not found anything definite yet that says), but from purchased books I have the new SQLite files rather than old JSON.

I had to edit in the ASIN/book type (EBOK) into the mobi file, which metadata/converting/quality check in Calibre would usually take care of. I chose this one, which I think is right (it's also 75 cents on the store ).
What issues are you having with the sqlite dll?

The issue was mostly a combination of being a tired dimwit and not having used Windows for anything serious in a while

I'm on a Mac, so was running an almost new Win7 install under virtualbox. I was getting an error about the SQLite DLL or its dependencies not being found. Some googling led me to a possible solution (from the homepage of the SQLite DLL) but none of that worked, was a total red herring, paired with my Vbox install's Windows Update service playing silly buggers.

Simple answer: A good sleep and a large coffee made it clear that installing Microsoft Visual C++ 2010 x86 redistributable was the required dependency. Works fine now

I imagine any regular Windows user would have this, but might be worth listing as a dependency for anyone in my situation.

The Calibre plugin does a decent job, but I don't think it has been updated in a while; I don't use it so I don't check up on mj's progress. However, the newest version of this app gives you more features (notable clips, end actions, start actions, etc).

I like the idea of the plugin more (perhaps just because I'm not a Windows user these days!), but I think your App is so far ahead in terms of features and polish. I'll get in contact with the plugin's author to see what his plans are - I'm happy to help out with that.

From your side - is there a reason why the App is preferable to the plugin? One obvious difference is language - perhaps doing this in .NET is simpler somehow or has better existing libraries..

Edit: Attached is a new-format X-Ray along with the End Actions, Start Actions, and Author Profile. I included the aliases/chapter files that I used for building as well.
I included the copy of the book I used; for some reason that I can't explain, the X-Ray button wouldn't show up for me on the version I downloaded, despite having added the proper metadata. This one has been run through Calibre's mobi->mobi conversion with (hopefully) minimal visual difference. The X-Ray won't work properly unless you use this copy with it.
Note: This book is freely distributable in the US under the Project Gutenberg License.

Perfect - and comparing the output from plugin vs. app (and copying to Kindle) shows what I suspected - the plugin's X-ray character location calculations are completely off. The App is working them out perfectly. I shall dig further when I have time to see what is wrong.

Thanks again,
Anthony

[stoduk]

There seems to be a minor bug in the StartActions.. file - the amazon page has the unicode left/right double quotes (U+201C and U+201D), which don't seem to be valid JSON for some reason. Not sure why, I thought any unicode was fine in JSON.

Both an online JSON validator and my Kindle barf on this though - escaping it with \ prefix doesn't help. Changing it to \" works fine though (ie. an escaped U+0022 double quote). Saving the whole file as UTF-16 hasn't helped.

I'd be intereted to know what the final fix is..

[Ephemerality]

Originally Posted by stoduk

Not sure what the cut off is for old vs. new format (I imagine it is software version as much as device, but I've not found anything definite yet that says)

Yeah, it's mainly just firmware version 5.6+ that uses the new format.

Originally Posted by stoduk

Simple answer: A good sleep and a large coffee made it clear that installing Microsoft Visual C++ 2010 x86 redistributable was the required dependency.

Ah, that would do it. I'll add it to the list just as a precaution. Any specific error it was getting that might alert people?

Originally Posted by stoduk

From your side - is there a reason why the App is preferable to the plugin? One obvious difference is language - perhaps doing this in .NET is simpler somehow or has better existing libraries..

C# is my go-to language because of work, it's what I'm used to. I've never really liked Python, so I wasn't keen on making a plugin (it was awesome that mjwilson went ahead and did). The app is also a lot faster and I like having full control over what goes on within it.
I considered re-writing it all in C++ so it would be cross-platform, but haven't had the time.

Originally Posted by stoduk

There seems to be a minor bug in the StartActions.. file - the amazon page has the unicode left/right double quotes (U+201C and U+201D), which don't seem to be valid JSON for some reason. Not sure why, I thought any unicode was fine in JSON.

Both an online JSON validator and my Kindle barf on this though - escaping it with \ prefix doesn't help. Changing it to \" works fine though (ie. an escaped U+0022 double quote). Saving the whole file as UTF-16 hasn't helped.

Hm... What does it do on the Kindle (I don't have a device to test atm)? The Amazon StartActions files are in UTF-8 so I would have assumed it would display fine... You could try \u201c and \u201d. If that works, we may have to do some more cleanup on the strings before writing them...

[Ephemerality]

Originally Posted by stoduk

I'd be interested to know what the final fix is..

Just as an update, we're working on a fix now. It just converts any unicode characters into "\u1234" format, which should work.

[stoduk]

Hi,

Sorry for slowness, had some family stuff to deal with, so this got put on the backburner. Inline..

Originally Posted by Ephemerality

Yeah, it's mainly just firmware version 5.6+ that uses the new format.

OK, thanks. That makes it easy to decide if a connected kindle supports new or old, as the version is available.

Ah, that would do it. I'll add it to the list just as a precaution. Any specific error it was getting that might alert people?

Yay for search histories: "System.IO.FileNotFoundException: Could not load file or assembly 'System.Data.SQLite.dll' or one of its dependencies. The specified module could not be found."

C# is my go-to language because of work, it's what I'm used to. I've never really liked Python, so I wasn't keen on making a plugin (it was awesome that mjwilson went ahead and did). The app is also a lot faster and I like having full control over what goes on within it.
I considered re-writing it all in C++ so it would be cross-platform, but haven't had the time.

Python seems quite marmitey!

Have you done any measurements to quanitfy any speed difference? I've found one problem with searching in the plugin that should speed it up, but overall for me the more obvious slow part is fetching the shelfari page (not sure how much to blame on shelfari, and how much to blame on sky broadband). For testing I'm caching the page locally, which makes things much quicker.

Hm... What does it do on the Kindle (I don't have a device to test atm)? The Amazon StartActions files are in UTF-8 so I would have assumed it would display fine... You could try \u201c and \u201d. If that works, we may have to do some more cleanup on the strings before writing them...

It would just not load (ie. greyed out X-Ray button). Sounds from your latest post that you fixed it, which is good.

[Ephemerality]

Originally Posted by stoduk

Yay for search histories: "System.IO.FileNotFoundException: Could not load file or assembly 'System.Data.SQLite.dll' or one of its dependencies. The specified module could not be found."

Weird, that's usually the error when you didn't extract the dll when you got the app. I'm assuming it was unzipped and in place?

Originally Posted by stoduk

Have you done any measurements to quanitfy any speed difference? I've found one problem with searching in the plugin that should speed it up, but overall for me the more obvious slow part is fetching the shelfari page (not sure how much to blame on shelfari, and how much to blame on sky broadband). For testing I'm caching the page locally, which makes things much quicker.

No measurements were done, but when I was testing with Song of Ice and Fire books, the app was pretty quick and the plugin was ungodly slow. The slowness from Shelfari seems normal for whatever reason (coming from Canada with a quick connection). Caching seems like a good idea, how long are you keeping the cached content?

Originally Posted by stoduk

It would just not load (ie. greyed out X-Ray button). Sounds from your latest post that you fixed it, which is good

Oh, the problem I was looking at was in the StartActions (menu button, About This Book). I think I've got it worked out either way. X-Rays themselves should work fine with unicode characters.