X-Ray Builder - Windows Application

[Ephemerality]

Originally Posted by juan_carlos

In this case you have to manually correct every single name. I think it tries to split the left side, and finds duplicates because it contains the same name twice - Lini (aka Lini Eltring). So delete one of the instances, and also 'aka' if you want to be totally correct.

Can't do that or it just won't match those terms at all. The left side needs to match exactly with what was downloaded from Shelfari. There could be an issue with the commas in the name though, I'll have to double-check that when I'm off work.

Originally Posted by cant_feel

If I try to auto split then I get this error

Spoiler:

Attempting to build X-Ray...
Spoilers: Disabled
Offset: 0
Downloading Shelfari page...
Shelfari URL: http://www.shelfari.com/books/25411/The-Path-of-Daggers
Duplicate term "Cha Faile" found. Ignoring this duplicate.
Duplicate term "Edarra" found. Ignoring this duplicate.
Duplicate term "Shaido" found. Ignoring this duplicate.
Duplicate term "Kumira" found. Ignoring this duplicate.
Duplicate term "Varek" found. Ignoring this duplicate.
An error occurred while creating the new X-Ray database. Is it opened in another program?
Length cannot be less than zero.
Parameter name: length

On the other hand this is the Aliases I get from the shelfari

Spoiler:

Rand (aka Rand al'Thor, Car'a'carn, Dragon Reborn, Chosen One, Coramoor)|
Mat (aka Matrim Cauthon)|
Perrin (aka Perrin t'Bashere Aybara)|
Faile (aka Faile ni Bashere t'Aybara)|
Moiraine (aka Moiraine Damodred)|
Elayne (aka Elayne Trakand)|
Nynaeve (aka Nynaeve ti al'Meara Mandragoran)|
Birgitte Silverbow|
Alise (aka Alise Tenjile)|
Bryne (aka Gareth Bryne)|
Flinn (aka Damer Flinn)|
Maighdin (aka Maighdin Dorlain)|
Dashiva (aka Corlan Dashiva)|
Lini (aka Lini Eltring)|
Aes Sedai Aviendha|
Egwene (aka Egwene al'Vere)|
Reanne (aka Reanne Corly)|
Masema (aka Masema Dagar)|
Galina Casban|
Romanda (aka Romanda Cassin)|
Weiramon (aka Weiramon Saniago)|
Aram|

So stuff only on the left side, nothing on the right.

In the default alias file, you'll *never* get anything on the right of the | symbol; only if you make it yourself or if one was available on my server (aliases don't come from Shelfari at all).
In the case of this book, where there's so many (aka whatever)'s in the names, the automatic split won't work (which is what caused that error). The duplicate errors you can usually ignore.
A lot of the issues on this one are caused by the person who edited Shelfari and decided that having 'Nickname (aka THEIR FULL NAME)' for every single character was a good idea...
If you go through and edit the aliases to look more like this (feel free to copy/paste this bit as a starting point), it should work:

Rand (aka Rand al'Thor, Car'a'carn, Dragon Reborn, Chosen One, Coramoor)|Rand al'Thor,Car'a'carn,Dragon Reborn,Chosen One,Rand
Mat (aka Matrim Cauthon)|Matrim Cauthon,Mat
Perrin (aka Perrin t'Bashere Aybara)|Perrin t'Bashere Aybara,Perrin

You should turn off the automatic split as well.
If I get a chance later, I'll throw one together and put it on the server anyway.

Originally Posted by juan_carlos

It seems to work perfectly with -16. The highlights are still misplaced in the x-ray menu, but in the actual text it works.

Huh, interesting. How far off are the highlights? The offset might be different for yours (making it that much more difficult to figure out lol).

[juan_carlos]

Originally Posted by Ephemerality

Huh, interesting. How far off are the highlights? The offset might be different for yours (making it that much more difficult to figure out lol).

It is kind of random how it highlights the words, sometimes in the middle of pages, but the names are always highlighted correctly. I'll make a screenshot later.

[Ephemerality]

Originally Posted by juan_carlos

It is kind of random how it highlights the words, sometimes in the middle of pages, but the names are always highlighted correctly. I'll make a screenshot later.

As long as the name-highlights are good, that's great. Where the -16 shift is coming from, I still have no idea, but if it's static for everyone, that's progress at least.
The randomness of the excerpts could be due to the shortening process. If a character shows up too far within a paragraph to display, the builder will cut off the excerpt roughly around where the character is, depending on where sentences end. I don't think it does much for the start of the excerpt though.
I haven't had a look at that portion in a while, I'll have to go through it and see if it can be improved easily at all...

Edit: Now that I think of it, the entire reason I added the 'offset' option to the original command-line version was for Amazon books that didn't have X-Ray support yet. I pulled them off my Kindle, removed DRM, created X-Rays for them, then loaded the X-Rays alongside the original DRM-protected copy. I always attributed the offset to the DRM-removal process, but it looks like it's AZW3 as a whole.
I've been scouring through the mobi format documentation and the unpacking process to see if I can figure out where the 16 character offset is coming from, but no luck so far....
If -16 works for *everyone* who tries it, I may just have to make it automatically use that value for AZW3s unless another solution is found.

[Ephemerality]

Originally Posted by ratanparai

After enabling "Use kindleUnpack", which was disabled by default, android kindle app is now showing X-ray information for some reason without kindleUnpack the generated uuid was invalid/wrong. Because the only thing that is changed is the xray file name's hexadecimal value at the end of the file name.

Just as an update on this, I found the bug that was causing it. It will be fixed in the next version.

[Marauder1]

Table of contents including too much text.
The answer to my question may be available, but I have not found it.

When I run x-ray builder, it adds essentially all of the text into the chapter heading in the table of contents. The original only includes the actual chapter titles, but after the build, the table of contents is larger due to the added text.

What am I doing wrong?

[Ephemerality]

Originally Posted by Marauder1

Table of contents including too much text.
The answer to my question may be available, but I have not found it.

When I run x-ray builder, it adds essentially all of the text into the chapter heading in the table of contents. The original only includes the actual chapter titles, but after the build, the table of contents is larger due to the added text.

What am I doing wrong?

It sounds like they are just being detected incorrectly.
You should be able to go into the ext folder and modify the chapter headings that were dumped to the chapters file.
If you want, you can post the build log here in a spoiler tag and pm the book to me so I can investigate and improve the chapter detection.

[Marauder1]

I changed some setting and the problem went away. Not sure what I did wrong.

Thanks for such a great utility!

[Ephemerality]

Originally Posted by Marauder1

I changed some setting and the problem went away. Not sure what I did wrong.

Thanks for such a great utility!

Cool! Any idea which setting it was?

[steinchen]

Originally Posted by Ephemerality

• /c - Characters are usually case-insensitive, this will make it sensitive. Good for terms that might show up and falsely match in other contexts. e.g. a character named Summer would incorrectly match on 'summer'.
• Example: Khal Drogo|/i,Drogo

Hi, just tried v2.0.10 GUI and it's working great. But I have a little problem with case sensitive names on my PW3.
First I built an alias file without any parameters. The file worked but "Will" was suddenly the most important character in "A Game of Thrones". So I searched and found the /c parameter. Building the xray file still works but my kindle still shows every "Will" and "will", the same happens for "Ned" Stark and others.
Does the Kindle cache the definitions somewhere or is there maybe a problem with the case sensitive parameter? By the way, the preview in the builder shows the same number of mentions as below.
I'm using a .mobi file, as it didn't find any chapters in my .azw3.
Here are some examples:

• Alias: Eddard Stark|/c,Eddard,Ned
  Count of mentions: 1988
  Right mention: Lord Eddard Stark dismounted and his ward...
  Wrong mention: There are things to be learned even from the dead.
• Alias: Will|/c
  Count of mentions: 768
  Right mention: Will saw them
  Wrong mention: We will find these men.
• Alias: Robert Baratheon|/c,Robert,Rob
  Count of mentions: 764
  Right mention: "For Robert!" he shouted.
  Wrong mention: At least as his wardrobe was concerned.

And as a feature suggestion: What do you think about a switch to only search for full words? Usefull for short names like Ned or Arry (so it wouldn't find turned and parry).

[Ephemerality]

Yeah, sorry about that. The /c parameter is actually broken in 2.0.10, it will be fixed in .11. The /r parameter for regex searches also isn't available in .10. I'll PM you a copy of the new version, but it should be out later this week as well if I get a chance.
Realistically, it should only be looking for whole words by default. I'll look into making that change, hopefully without breaking anything.

[steinchen]

Thanks for the swift reply. In .11 the case-sensitive switch works. I already had a look at your github but as I use Mono on OS X I didn't finish optimizing it, yet. I'll continue with it now.
2.0.11 on Windows 7 works as it should.

EDIT: Now I see multiple matches for the same mention.
Example:
Robert Baratheon|/c,The Ursurper,Demon of the Trident,Robert,Rob
This Alias matches two times for Robert and three times for Robert Baratheon (I guess once for Robert Baratheon, once for Robert and once for Rob).
EDIT2: Reverted to commit 63e5ed6 and there are no multiple matches. I think the changes in e7590db (Scan each paragraph for every alias and match all instances) might match too much.
EDIT3: Just confirmed, the double/triple matches start at commit e7590db.
So except for the known "whole word"-issue (Robb matches for Robb and also Rob) the X-Ray from 63e5ed6 look pretty good to me.

[Ephemerality]

Are you actually able to get it working in Mono? I tried but ran into multiple issues that I didn't bother trying to fix.

The multiple-matches thing is new and still needs work; we were trying to make it so that every instance of the character will be clickable (to open their X-Ray profile), because that always seems to be the #1 complaint. Need to work on ignoring duplicate matches like that. The regex that helps match things like "Rob's," (the punctuation at the end needs to match as well for the clicking bit) needs some tweaking too, it shouldn't be matching Robb if it's supposed to be Rob.

[steinchen]

Originally Posted by Ephemerality

Are you actually able to get it working in Mono? I tried but ran into multiple issues that I didn't bother trying to fix.

At the moment I just build it in mono and run it on Win7. This works so far. Running on OS X has several issues which I might have a look at:

• Hard-coded paths with backslash: Fixed using Path.Combine
• Log-Windows doesn't update in real time: Didn't check yet, nothing really important
• Copy & Paste doesn't work: Known Mono on OSX problem
• Some error with SQLite: That's the point where I just copied the files to Win7

Your changes worked great. Rob and Robb aren't the same person anymore and the multi-metions are gone, too.

[Ephemerality]

Awesome.

Any reason you're using Mono instead of Visual Studio, if you have access to Windows?

[steinchen]

Just that Mono was already installed. As the compiled files run on Windows I didn't bother installing Visual Studio.
My long term goal would be porting your code to python to use it as a calibre plugin. But as I have an hour a day at most I didn't even finish reading and commenting your code.
I have to say that finding apostrophes doesn't work in the latest commit. Only noticed it last night while reading. In my current book they use ’ (U+2019, which according to ISO 10646 is the preferred apostrophe) for apostrophes and in your code it's a ' (U+0027). I guess some people might use ` (U+0060) or ´ (U+00B4), at least I did as a kid. Regex and I don't really get along but I tried changing

string punctuationMarks = @"(?(')'s?|')?\b[!\.?,""'\);]*";

to

string punctuationMarks = @"(?(['´`’])['´`’]s?|['´`’])?\b[!\.?,""'\);]*";

but this also didn't work. Then I tried

string punctuationMarks = @"(?(’)’s?|’)?\b[!\.?,""'\);]*";

hoping it would search for ’ but this also didn't work.
On the weekend I'll have a closer look at this (and I'll try to learn the regex syntax).