Invalid TOC links in some azw3 files

[KevinH]

Hi,
I'm out of town traveling and will fix it for KindleUnpack when I get back home unless DiapDealer beats me to it

KevinH

[darryl]

Thanks Kevin. Enjoy your trip.

[Hitch]

I still really want to know what the hell caused it.

Hitch

[darryl]

Originally Posted by Hitch

I still really want to know what the hell caused it.

Hitch

I'm probably the least qualified on this thread, but I'll give you my understanding of the situation.

I think:

1. Some of us assumed, incorrectly, that the .azw3 file did not have a working toc. In fact it did.
2. What we were looking at, of course, is not the actual files, but the "reconstruction" of those files by either Calibre or KindleUnpack.
3. Both programs were extracting an incorrect toc.ncx. This toc had links referring to the single html file without the necessary "fragments".
4. In fact, the azw3 file does contain fragment information for the links in the form of uuid's.
5. Neither Calibre nor KindleUnpack were extracting these uuid fragments from the azw3 file, because, to use Kovid's description of the changes made, "kindlegen produced azw3 files that do not use normal HTML anchors for linking". It seems that both programs were expecting the use of normal HTML anchors and hence could not reconstruct the toc correctly from the file.
6. I don't know precisely what a normal HTML anchor would be for this purpose, but clearly a simple <h1> tag does not qualify.
7. I don't know why kindlegen dealt with the situation in the way that it did. Perhaps the original input toc was invalid and it generated a new one based on the <h1> tags? With my limited knowledge and experience of this area I cannot even dismiss the possibility that this was happening with every .azw3 file where the input was a single html file, as I think you said in an earlier post, probably based on a word document.
8. Kovid's modifications mean that Calibre including the viewer now correctly deals with these fragments. Kevin, or perhaps Diap Dealer, will do the same for KindleUnpack, which will presumably result in KindleUnpack extracting a toc.ncx from these files including the uuid "fragment" information.

As I said, this is my understanding. I am quite possibly wrong in at least some particulars, and would appreciate any corrections by you or others here with a better understanding. I found the problem quite intriguing, and learnt a little about a few different areas, including the structure of .mobi and .azw3 files, KindleUnpack and of course TOC's.

regards,

Darryl

[kovidgoyal]

Links in an azw3 file are in the form of byte offsets into the raw html. In the past these byte offsets have always pointed to tags that have an id attribute. So calibre would simply use that id attribute as the anchor when converting the byte offset based link into a normal html link. Your problem file had byte offsets that point to tags with no id attribute. In this case calibre would simply point to the file, with no anchor.

The assumption that tags pointed to by byte offsets will always have ids is reasonable, since typically azw3 files are created from epub/html, where links always use ids. However, in the case of your file, the file was presumably created in a way that did not require ids. The links could have been created, for example, using XPath expressions, or some such. So in this case, with my fix, calibre now generates a unique id for the tag, when one is missing.

Oh and I should mention, that the azw3 input plugin in calibre is largely based on KevinH's original work reverse engineering the azw3 format.

[darryl]

Kovid. Thanks for taking the time to explain.

[DiapDealer]

Originally Posted by KevinH

Hi,
I'm out of town traveling and will fix it for KindleUnpack when I get back home unless DiapDealer beats me to it

I took a stab at it. I'm not going to make a new release or anything until Kevin gets back, but anyone's free to pull the latest KindleUnpack from github if they want to try it out on these "special" books.

I only had the one for testing, so...

https://github.com/kevinhendricks/KindleUnpack

[KevinH]

Hi DiapDealer,
Thanks! Looks good! I will make an official 0.81 release upon my return. From Kovid's description it almost sounds like someone passed an old single file mobi 6 as input into some new kindlegen version (remember the filepos internal link destinations) and it converted the filepos literally.

Take care,

KevinH

[DiapDealer]

I've been trying to duplicate it using Kindlegen/Previewer, but I just can't seem to make it happen. There's an update for KindlePreviewer that I'll check out next. My curiosity has been piqued.

[Hitch]

Originally Posted by DiapDealer

I've been trying to duplicate it using Kindlegen/Previewer, but I just can't seem to make it happen. There's an update for KindlePreviewer that I'll check out next. My curiosity has been piqued.

Me, too. How are you endeavoring to make it occur?

Hitch

[DiapDealer]

Originally Posted by Hitch

Me, too. How are you endeavoring to make it occur?

Hitch

At this point, I'm at the "throw everything I can think of at Kindle(gen|Previewer) and see what comes out" phase.

html, opf, epub, raw unpacked mobi markup...

[eschwartz]

KevinH's guess that it is an all-new-and-improved way of totally mangling mobi7 input sounds reasonable.

I wonder if KDP uploads are now being converted when people upload a calibre-converted mobi7... because they are absolutely insistent on using calibre for some totally inexplicable reason.
Amazon probably wants customers to actually get AZW3 if possible.

You may not be able to duplicate this with *just* kindlegen/previewer.

[Hitch]

Originally Posted by DiapDealer

At this point, I'm at the "throw everything I can think of at Kindle(gen|Previewer) and see what comes out" phase.

html, opf, epub, raw unpacked mobi markup...

Diap:

Did you try uploading a Word file at the KDP, and downloading a step-6 mobi?

Hitch

[DiapDealer]

I'm actually not KDP enabled.

[Hitch]

Originally Posted by DiapDealer

I'm actually not KDP enabled.

Uh, well....I think that the general consensus of utterly uninformed opinion is that the whatever-it-is is happening at the KDP, no? You wanna send me a file to upload there, and hand back to you?

I also have a Word file MOBI around here somewhere, that I made for a video demo, I think, but I made it about a year ago. I probably still have the source Word file, though, and could repeat the experiment. It was a PG copy of a Christie Tommy & Tuppence that I cleaned up and tweaked a bit. Whatcha want?

Hitch