Invalid TOC links in some azw3 files

[DiapDealer]

Interesting.

Even unpacking the raw markup language, doesn't reveal a working html toc. I suspect we're seeing something that KindleUnpack or calibre's unpacking routines hasn't had to deal with before.

The NCX could have been hosed from the get-go (not the first time that's happened on a Kindlebook), but I'm not sure how a working toc is being achieved (on Kindle devices/apps) when there doesn't seem to be any html toc anywhere.

[darryl]

Amazon will be devastated! A TOC that only works on Kindle devices and software? I'm sure they will rush to fix i!. If only we could look at the source code.

I had not considered that the problem may lie with unpacking. Presumably there may in fact be a working toc which is not being unpacked or unpacked properly. For what it is worth, my inexpert opinion on the most likely explanation for the working TOC is that we might be seeing the result of some type of TOC fallback algorithm in the code which operates in the absence of a toc.html.

[Hitch]

Originally Posted by eschwartz

Hitch -- as for the Go To menu, it has always shown individual chapters, for a given value of "whatever has been properly formatted into the navigational ToC.
The nav elements are scrollable and sort of appear in an embedded view in the Go To menu, almost like the toc.ncx is rendered in an iframe as its own Go To element. Guide elements appear above and below.

I have been made aware that Kindle Previewer shows stuff a little different from the way E-Ink Kindles do, but I figure you'd know this already, what with all the testing devices you own.

Snookums:

Well...on some of the e-inks, yes. But on Fires, no. Not a'tall. The Fire doesn't have a Go To (at least, none of those we have in-house do!) that displays the chapters. The Go To shows what's in the Guide, and the TOC is itself a separate entity on that. You click the TOC, go thither, and navigate from there. On our PPW and Voyage, yes, the Go To displays the Chapter #'s. I admit, as a reader, I had actually completely forgotten that the Go To shows the Chapters !

Oish. I mean, I must look at this 10+ x a week, in terms of checking books that need my own QA, and I'd utterly spaced it. Lord. (My only defense is, I am still so accustomed to looking at the OLDER e-inks, for the KF7 stuff, that I just FORGOT, aged old bag that I am.)

Anyway....back to the topic at-hand. Diap, how the HELL is that working, do you think? I wonder if we ought to experiment with a book we (my co) has made, to see if we see something similar??? Post-publication? I would have to ask around, and figure out what client didn't use DRM, of course, but... Whatcha think? At least we'd know what we'd started with?

Hitch

[Jeff L]

Could it be the NAV element?

The Kindle pub guide includes this now. Perhaps the unpack isn't translating it.

[Doitsu]

Originally Posted by Jeff L

Could it be the NAV element?

The Kindle pub guide includes this now. Perhaps the unpack isn't translating it.

No. I just tested it with a KindleGen compiled ePub3 file containing only a landmarks nav based TOC. KindleUnpack 0.7.7 had absolutely no problems extracting the file with the landmarks nav elements from the mobi7 and mobi8 files.

BTW, ePubCheck 3.0.1 won't display error messages if two nav elements point to the same target.
KindleGen displays a warning message, but no warning message at the end of the build log:

Warning(NCXBuilder):W30001: An overlapping subsequent nav point found for the nav point: "xxx"

IMHO, this is a bug, because Kindle ebook creators won't notice problems with nav points, unless they look for W30001 warnings.

@darryl: Unpack your book again with KindleUnpack 0.7.7. Check the Write Raw Data option and search the .rawml file for <nav ...>...</nav> tags.

[Notjohn]

Originally Posted by DiapDealer

The NCX could have been hosed from the get-go (not the first time that's happened on a Kindlebook), but I'm not sure how a working toc is being achieved (on Kindle devices/apps) when there doesn't seem to be any html toc anywhere.

Hosing the NCX -- and perhaps also the OPF file? Might that explain why the Start Reading Location jumps around so much?

Last week I published a new version of a book that has been up a couple of years. In the past, it always opened in the middle of the TOC, as many of my books have done. This time, without any change in the Sigil semantics, it opened at the title page.

I quick went back to the Sigil epub and inspected it. Sure enough, I had labeled the title page as Text, and it showed properly in the Guide section. (Which I hadn't changed at all from the original version.)

[darryl]

Originally Posted by Doitsu

@darryl: Unpack your book again with KindleUnpack 0.7.7. Check the Write Raw Data option and search the .rawml file for <nav ...>...</nav> tags.

Good thought. But it looks like the .rawml file contains no such tags.

I updated my KindleUnpack to 0.7.7. Unpacked with raw option. Ran following commands:

grep "<nav" Exodus\ \(The\ Exodus\ Trilogy\ Book\ 1\).rawml

grep "</nav" Exodus\ \(The\ Exodus\ Trilogy\ Book\ 1\).rawml

Nothing found.

A quite annoying little mystery.

[DiapDealer]

Originally Posted by Notjohn

Hosing the NCX -- and perhaps also the OPF file? Might that explain why the Start Reading Location jumps around so much?

Last week I published a new version of a book that has been up a couple of years. In the past, it always opened in the middle of the TOC, as many of my books have done. This time, without any change in the Sigil semantics, it opened at the title page.

I quick went back to the Sigil epub and inspected it. Sure enough, I had labeled the title page as Text, and it showed properly in the Guide section. (Which I hadn't changed at all from the original version.)

It doesn't explain why neither KindleUnpack nor calibre can extract either a working html toc or a working ncx from a kindlebook whose Go To sections clearly functions properly (the entire toc is available--and works in Kindle for PC. After extraction, one or the other (ncx, or html toc) should be valid/working.

I'm not ruling out something wrong with the Kindlebook, but the simple fact is that if the toc works in the Kindlebook ... then KU/calibre should be able to extract a working version of it.

Either something has changed on the Kindlegen end, or this book contains something that KU/calibre has never encountered before (or both).

[darryl]

Originally Posted by DiapDealer

Either something has changed on the Kindlegen end, or this book contains something that KU/calibre has never encountered before (or both).

Don't forget that this book is not a one-off. Based on very small numbers at this stage, as many as 10% of Kindle books could have similar problems, though I doubt it is this high. I think the problem may have been there for a while but has gone undetected. This is chiefly because it affects a relatively small number of books and no one reading on a Kindle or Kindle software would notice any problem. Nor would anyone reading on other devices or software including the Calibre Viewer notice unless they had cause to use the TOC links. Anyone converting to epub and using Moon Reader cannot even read more than the first few pages unless the TOC is deleted or corrected, a known problem.

[kovidgoyal]

I will be happy to take a look at one of these books (preferably a small one).

[eschwartz]

PMed you with a copy of darryl's example ebook.

[kovidgoyal]

https://github.com/kovidgoyal/calibr...caacff008c0f6f

[eschwartz]

Thanks Kovid!

I pulled the changes, and the book now unpacks into the editor with a UUID for a filename fragment.

Someone should think of doing the same in KindleUnpack.

[darryl]

Thanks Kovid. A comprehensive solution very quickly. I probably should have brought it to your attention in a private bug report.

It seems the mystery is solved. The TOC is produced from lines like the following. I don't know exactly what the aid= tags do, though they seem to be sequentially numbered as used. But then in the second example, chapter 1, the use of 1G following 1F is strange. I had thought the numbers were hexadecimal.

<h1 style="margin:0.00% 0.00%; page-break-before:always; line-height:150%; " aid="12"><span style=" font-size:1.2rem; font-weight:bold" aid="13">Prologue</span></h1>

<h1 style="margin:0.00% 0.00%; page-break-before:always; line-height:150%; " aid="1G"><span style=" font-size:1.2rem; font-weight:bold" aid="1H">Chapter 1</span></h1>

In any case, it seems to be fixed. Calibre now creates a working toc.ncx and links work in calibre viewer. As eschwartz says, KindleUnpack now needs corresponding alterations.

Thanks to everyone who took an interest.

[kovidgoyal]

You're welcome

FYI, aids are base32 not base16