Hacks X-Ray file generator for Kindle Touch

[RobbieClarken]

Originally Posted by Ephemerality

I've actually been working quite a bit over the past week on creating an x-ray builder that generates all of the loc and chapter information by unpacking the mobi file and running through the raw book content (the page-by-page info is calculated on the Kindle itself).
The format is fairly simple, each loc points to a point in the book and defines the excerpts you see on that character's page. The locations are positions within the raw html file (I've been unpacking the mobis with mobi_unpack using the -r flag to get the raw markup).

Thanks for the info Ephemerality! I've been working on writing a python script to generate the xray file and I've run into a problem.

I have generated the raw markup language with kindleunpack.py but when I compare the locs from the xray file they don't exactly match up with the positions in the rawml file.

For one book, each loc is 14 characters off the position in the rawml file. For another book, the discrepancy ranges from 22 to 2000 characters.

Have you come across this? (I get the same issue using the version of mobi_unpack you link to.)

[Ephemerality]

Originally Posted by RobbieClarken

For one book, each loc is 14 characters off the position in the rawml file. For another book, the discrepancy ranges from 22 to 2000 characters.

Have you come across this? (I get the same issue using the version of mobi_unpack you link to.)

Good question. I had the same issue, but only on DRM'd books. So if I had the DRM'd version on my Kindle, transfer it to my pc and remove the DRM to create the X-Ray, then use the X-Ray on the original DRM copy it would have an arbitrary offset like you're describing.
I have been copying to pc, removing DRM, then using the DRM-free version on my Kindle.

Edit: Also, I ran into issues with the locs being off because of the encoding being used. The books I've used are using UTF8, which causes things like quotes to take up more space than normal (I think 3 characters instead of 1). So if you're not loading the book and taking that into account, you will end up counting those characters as 1 loc instead of 3... hopefully that makes sense.

[Ephemerality]

Originally Posted by raist1976

I have one question, what does the original script and your program (as weel as the Calibre Plugin) fetch from Shelfari ? The description without spoilers or with spoilers ?

I have added the feature, however I have not extensively tested it yet. A new version has been pushed to the repo and I will attach a new binary to my original post (#145).
You can enable the use of spoilers with the --spoilers token.

[raist1976]

Originally Posted by Ephemerality

I have added the feature, however I have not extensively tested it yet. A new version has been pushed to the repo and I will attach a new binary to my original post (#145).
You can enable the use of spoilers with the --spoilers token.

Nice ! But I don't know if you tested the existence of Spoiler description ? Best way would be, if no spoiler description found, then use the spoiler free description.

[Ephemerality]

Originally Posted by raist1976

Nice ! But I don't know if you tested the existence of Spoiler description ? Best way would be, if no spoiler description found, then use the spoiler free description.

Shelfari does all the work already in that regard. If there is no spoiler description available they only show the spoiler-free one (same as if you're looking at the website yourself).

Also, as rikkia found out, there is an issue running mobi_unpack.py if you do not have Python installed or if you have it installed but do not have the Python directory included in the Windows PATH var.
I have updated with an option (-p) to specify this location. It will still default to using the original "python" command. You can test whether it works by opening a command prompt and typing "python".
Download uploaded on original post (#145)

Originally Posted by raist1976

But I wonted how big the except could be ?

Almost forgot about this. From the looks of things, the description can be ~550-560 characters or so (again, it depends on how much space it takes up due to the font, not the # of characters directly).
If things work properly, if a description is too long you can click on the Shelfari link to be taken directly to the character/term's Shelfari page which includes all the books they are in and the full descriptions from each.

[Ephemerality]

To whomever downloaded v1.11, please grab v1.12 instead. I forgot to remove the URL I was using to test the spoiler code so it will always download from that link. Sorry!

[twobob]

Done . thanks

[Ephemerality]

Updated to v1.13. (#145)
It now includes a py2exe packed version of mobi_unpack.py, eliminating the need to have Python installed (again, I think this is ok?).

I have not had a chance to test this on more than my two Win7 PCs, so let me know if this totally fails.

If you have already saved a mobi_unpack location as the default, you can use the new exe by specifying "-u dist/mobi_unpack.exe". You can still use Python and mobi_unpack.py like before if you wish!

Mobi2mobi is not needed anymore either.

In theory, you should be able to download the binary and start drag-dropping mobi/azw files onto it without worrying about the pre-requisites!

[twobob]

Thanks. : )

quick question....

Usage: xraybuilder [-o path] [--offset N] [-p] [-r] [-s shelfariURL] [--spoilers] [-u path] mobiPath
-o path (--outdir)	Path defines the output directory
			If not specified, uses ./out
--offset N	Specifies an offset to be applied to every book location.
			N must be a number (usually negative)
			See README for more info
-p path (--python)	Path must point to python.exe
			If not specified, uses the command "python",
			which requires the Python directory to be defined in
			the PATH environment variable.
-r (--saveraw)		Save raw book markup to the output directory
-s (--shelfari)		Shelfari URL
			If not specified, there will be a prompt asking for it
--spoilers		Use descriptions that contain spoilers
			Default behaviour is to use spoiler-free descriptions.
-u path (--unpack)	Path must point to mobi_unpack.py
			If not specified, searches in the current directory

After used once, mobi2mobi and mobi_unpack paths will be saved as default and are not necessary to include every time.
You can also drag and drop a number of mobi files onto the exe after those paths have been saved.

Books must be DRM-free. If you create an X-Ray file for a book, but still use the DRM copy on your Kindle,
you will run into issues. The X-Ray has to be used with the DRM-free version it was created from.

is the usage outdated now. I see it has changed a bit (I think)

much obliged

[twobob]

v 1.3 via Wine..

(and yes I am trying stupid things)

I found one of Ians that someone had already done some work on... And wasnt already xray enabled

me@dev ~/.wine/drive_c/Program Files/xray $ ./XRayBuilder.exe -s http://www.shelfari.com/books/18023565/Third-Sight -o /home/me/books ./3rd\ Sight_\ A\ Horror\ Novel_B00AJ7FC1Q.azw3

Using /home/me/books as an output directory.
Using default Python command. Ensure Python's directory is included in your PATH environment variable.
Processing 3rd Sight_ A Horror Novel_B00AJ7FC1Q.azw3...
Running mobi_unpack to get book data...
Error trying to launch mobi_unpack, skipping this book. (ApplicationName='dist/mobi_unpack.exe', CommandLine='-r -d "./3rd Sight_ A Horror Novel_B00AJ7FC1Q.azw3" "/tmp/x8wtiyq2.y0d"', CurrentDirectory='')
All files processed! Press Enter to exit.

hmm..

[Ephemerality]

Originally Posted by twobob

is the usage outdated now. I see it has changed a bit (I think)

Yes, I suppose it should be updated a bit!
You can still specify the locations to mobi_unpack and python (if you point it to mobi_unpack, python needs to be specified or saved already), but otherwise it will try to use dist/mobi_unpack.exe.

The new offset option is for DRM-enabled books, since the excerpts are normally offset by a certain amount when you open the X-Ray on the Kindle.

The other options are pretty much the same.

v 1.3 via Wine..

(and yes I am trying stupid things)

It looks like it couldn't find the mobi_unpack application to execute it, which is odd because if it got that far it means that it already checked to ensure the file does exist.
Can you try adding -u and the full path to the mobi_unpack.exe? If that works, I'll make it default to the full path instead of the short path its using now.

[twobob]

kk will do in a bit then

thanks for the feedback

[rikkia]

Not been able to reply sooner but for me V1.13 works perfect thanks for the work put into making it Ephemerality.

Unfortunately none of the books I have in my small library work. of 11 books 2 produce a x-ray file 9 crash the builder out usually failing at the unpack stage. The 2 xray files when loaded into the paperwhite give the "X-Ray is not available for the book" message with the centred "WG" at the bottom of the message window. I'll keep abreast of the thread just in case anyone manages to find a working solution to my issues

Rikki

[twobob]

me@dev ~/.wine/drive_c/Program Files/xray/dist $ chmod a+x ./mobi_unpack.exe

Duh!

me@dev ~/.wine/drive_c/Program Files/xray $ ./XRayBuilder.exe -u /home/me/.wine/drive_c/Program\ Files/xray/dist/mobi_unpack.exe -s http://www.shelfari.com/books/18023565/Third-Sight -o /home/me/books ./3rd\ Sight_\ A\ Horror\ Novel_B00AJ7FC1Q.azw3

Using /home/me/books as an output directory.
Using default Python command. Ensure Python's directory is included in your PATH environment variable.
Processing 3rd Sight_ A Horror Novel_B00AJ7FC1Q.azw3...
Running mobi_unpack to get book data...
Error unpacking mobi file: MobiUnpack 0.45
Copyright (c) 2009 Charles M. Hannum <root@ihack.net>
With Additions by P. Durrant, K. Hendricks, S. Siebert, fandrieu, DiapDealer, nickredding.
Error: first parameter must be a Kindle/Mobipocket ebook or a Kindle/Print Replica ebook.

Press Enter to exit.

hmm... looks close..

I'll attempt to get/make a mobi version?

[Ephemerality]

Originally Posted by twobob

me@dev ~/.wine/drive_c/Program Files/xray/dist $ chmod a+x ./mobi_unpack.exe

Duh!

me@dev ~/.wine/drive_c/Program Files/xray $ ./XRayBuilder.exe -u /home/me/.wine/drive_c/Program\ Files/xray/dist/mobi_unpack.exe -s http://www.shelfari.com/books/18023565/Third-Sight -o /home/me/books ./3rd\ Sight_\ A\ Horror\ Novel_B00AJ7FC1Q.azw3
hmm... looks close..
I'll attempt to get/make a mobi version?

Ah yes, execute permissions would be important

That's definitely a start there. It should be able to process KF8 file, the error you got there was from mobi_unpack, it may just be unable to find the book file. You may have to specify the full path to the book on the cmd line a well! If I had a GUI setup for it, that would probably be less of issue. I could probably change it to expand to the full path on its own if only the local path is given (since mobi_unpack is in a different folder). I'll take a look at it when off work.
If it still gives a similar error you could try a mobi file, but I think it's just not finding the file.
Thanks for testing it out for me!

Originally Posted by rikkia

Not been able to reply sooner but for me V1.13 works perfect thanks for the work put into making it Ephemerality.

Unfortunately none of the books I have in my small library work. of 11 books 2 produce a x-ray file 9 crash the builder out usually failing at the unpack stage. The 2 xray files when loaded into the paperwhite give the "X-Ray is not available for the book" message with the centred "WG" at the bottom of the message window. I'll keep abreast of the thread just in case anyone manages to find a working solution to my issues

Rikki

Thanks for giving it a shot, it's tough to debug things when it works fine in my environment!
Could you PM me a log of the program when it crashes and I can try to identify where the issue is, as well as a log of the successful build that gives the WG error on the Kindle?
The WG error refers to the GUID not matching what the Kindle is expecting. I ran into on only 1 of my books; in that case, the name Calibre had assigned to it was too long. I shortened the title in Calibre and it worked fine after that. You could also manually modify the database name of the book using mobi2mobi (mobi2mobi --databasename "New Name" "Path to file" --outfile "Path to file").