Real Page Numbers

[theducks]

Originally Posted by rpspringuel

So, it seems my understanding of copyright law was flawed and my test book was still in copyright. Sorry about that. I'll go looking for something that's out of copyright and create a new test book for those interested in testing.

I don't know if your understanding of the law was flawed or not.
MR does not even allow posting of 'Free' books that are under copyright (eg. Baen free Library)
We just try and abide by the owners rules

[rpspringuel]

If anyone is interested in testing, I've now uploaded a new test book which I believe to be out of copyright. I've attached it to the same post where I attached to code so that you can get both in one place.

[KevinH]

Hi,

A recent version of KindleUnpack has added support for decoding the "PAGE" sections in each mobi generated by kindlegen when an epub with either a pagelist in ncx or page-map.xml is provided for input.

This PAGE section is nearly identical to the APNX format. In fact, if you add 8 bytes of padding to the front of an APNX file contents you can use the same code to decode both.

This code has support for decoding multiple page naming schemes (arabic, roman, characters, etc).

KindleUnpack uses this code to recreate a page-map.xml file from kindlegen generated mobis (for just the mobi 8 part). The latest version of KindleUnpack will also allow the user to pass in an associated apnx file (associated with a mobi 8 (azw3) file and have it decoded.

FWIW, after removing all of the cruft just used to properly parse the command line into working utf-8, and removing the addition of 8 bytes of padding, this code could be easily modified and added to calibre to process the PAGE sections from kindlegen generated mobis.

I have attached it just in case it is of interest to you or Kovid.

Take care,

KevinH

[kovidgoyal]

@KevinH: Do any actual Kindles make us of the PAGE section in AZW3 files?

[KevinH]

Hi Kovid,

Only indirectly. Since real Kindles get only one version (Mobi 8 or Old Mobi) and not both upon download, the PAGE section is stripped out to become the actual APNX file after very slight modifications to add the palm database name and the asin.

To prove this theory, I used the actual APNX file from a recent purchase of the book Blood Rites by Jim Butcher, that I had downloaded to my Kindle for Mac (ie. an azw3 file and its associated apnx) and when I unpacked it the apnx offsets pointed EXACTLY to the <a id="page-X"></a> tags in the assembled text (raw text after inserting the fragments into their skeletons in the right place but before changing any links or anything else like that).

I then used kindlegen on the unpacked mobi (epub) and it recreated the exact same offsets in its PAGE sections in the mobi 8 part of the file.

So the original epub had either a pagelist in the ncx or a page-map.xml in it that referenced those id tags to mark the start of that page.

I also compared it to my hard copy of that exact same page and again they matched exactly.

So the PAGE sections created by Kindlegen are used to make the associated apnx file (with only addition of the proper metadata) and they will point exactly to the id tags used in the epub in either the pagelist of the ncx or the page-map.xml if either of those are used

Obviously, if they are just made-up offsets, none of this would have worked.

So you can think of the Kindlegen generated PAGE sections as keeping all of the relevant page information if and only if the input epub had either a page-map.xml (the Adobe standard) or pagelist info in the ncx. Otherwise kindlegen won't bother to create the PAGE section.

An apnx file on the other hand could be either just a set of page sized offsets or an exact page information taken from a PAGE section. There is no way to tell unless you look for the id= tags that marked the original pages in the epub it all was created from.


KevinH

Originally Posted by kovidgoyal

@KevinH: Do any actual Kindles make us of the PAGE section in AZW3 files?

[KevinH]

Hi Kovid,

One other thing. KindleUnpack can split Kindlegen generated dual/joint mobis into old mobi and azw3 pieces. Right now, it will strip out the PAGE sections (they are one of the sections pointed to by the SRCS offset) when it strips out the SRCS.

If you do decide to add support for PAGE sections in calibre, I can modify that code to leave the PAGE section in place for calibre to see and use. So calibre would have true page information from both kindlegen generated joint mobis, and azw3s split by KindleUnpack.

If you decide not to incorporate it, I can instead have Kindleunpack write out an apnx file directly (after adding in the required palm database name and asin), so that the true page information is not lost on the mobi side.

Just let me know which you prefer.

Take care,

KevinH

[JSWolf]

Originally Posted by KevinH

Hi Kovid,

One other thing. KindleUnpack can split Kindlegen generated dual/joint mobis into old mobi and azw3 pieces. Right now, it will strip out the PAGE sections (they are one of the sections pointed to by the SRCS offset) when it strips out the SRCS.

If you do decide to add support for PAGE sections in calibre, I can modify that code to leave the PAGE section in place for calibre to see and use. So calibre would have true page information from both kindlegen generated joint mobis, and azw3s split by KindleUnpack.

If you decide not to incorporate it, I can instead have Kindleunpack write out an apnx file directly (after adding in the required palm database name and asin), so that the true page information is not lost on the mobi side.

Just let me know which you prefer.

Take care,

KevinH

Please leave the PAGE section so it can be converted to ePub and work with ADE. I don't know if the ePub code will need modifying or not, but it would be good to keep it in so it can be fixed to work with ADE.

[kovidgoyal]

I'm somewhat unconvinced for the need for PAGE sections so far. It seems to me:

1) The only thing that uses it is Amazon's servers. And since you cannot send AZW3 by email anyway, generating AZW3s with PAGE sections appears to be useless, unless you intend to send you book to KDP, and KDP does not accept calibre generated AZW3 files anyway.

2) The other possible use case is converting kindlegen produced AZW3s to EPUB. However, since the only way to (typically) get hold of kindlegen produced AZW3s is generating them yourself, you should already have an epub with a page map, so I dont see why you would not want to use the source epub directly.

That said, the code is pretty trivial, so I might just add it anyway. But it is not a priority.

[KevinH]

Hi,
The other use case is distributing free of charge kindlegen generated mobis without the source zip archive. Kindlegen can in fact generate these now with a simple command line option and their terms of use allow you to give them away for free to others without going through Amazon.

If someone had such a kindlegen generated mobi and they could decide they would like to convert it to epub for reading on another device. If they use calibre to do this now, they would lose the real page information.

Similarly, with KindleUnpack you can split out the source and create azw3s with page info in PAGE sections, and post them for others to convert.

The nice thing about the PAGE section is that it keeps the real page info right inside the mobi where it belongs, and not in some external APNX file.

Either way, I plan to make KindleUnpack keep the PAGE section when splitting while adding the capability to create an APNX file from it too. Perhaps a standalone APNX file generator if that turns out to be useful.

Take care,

KevinH

Originally Posted by kovidgoyal

I'm somewhat unconvinced for the need for PAGE sections so far. It seems to me:

1) The only thing that uses it is Amazon's servers. And since you cannot send AZW3 by email anyway, generating AZW3s with PAGE sections appears to be useless, unless you intend to send you book to KDP, and KDP does not accept calibre generated AZW3 files anyway.

2) The other possible use case is converting kindlegen produced AZW3s to EPUB. However, since the only way to (typically) get hold of kindlegen produced AZW3s is generating them yourself, you should already have an epub with a page map, so I dont see why you would not want to use the source epub directly.

That said, the code is pretty trivial, so I might just add it anyway. But it is not a priority.

[kovidgoyal]

But anyone that's distributing kindlegen produced files would most likely also be distributing the source epubs.

In any case, as I said, I have no objections to including PAGE support into calibre, it's just not much of a priority, since the use case for it is rather minimal.

[KevinH]

Hi Kovid,

Understood. I have now modified KindleUnpack to read the PAGE sections and create the proper APNX on the fly (for both old mobi and mobi8 parts) which should hold most people until you get around to adding that feature to calibre.

Thanks!

KevinH

[JSWolf]

Originally Posted by KevinH

Hi Kovid,

Understood. I have now modified KindleUnpack to read the PAGE sections and create the proper APNX on the fly (for both old mobi and mobi8 parts) which should hold most people until you get around to adding that feature to calibre.

Thanks!

KevinH

But what about when the AZW3 is converted to ePub? Do the page numbers get kept?

[KevinH]

Originally Posted by JSWolf

But what about when the AZW3 is converted to ePub? Do the page numbers get kept?

That was the whole point of the change. If you have a joint mobi or azw3 with a PAGE section, then using experimental KindleUnpack_v072d will create an epub with a page-map.xml and the appropriate entries in the spine and manifest to that the epub will have the page numbers.

It will also allow you to pass in an existing APNX and **IF** it was generated from an epub with real page id= anchors to mark page start points, then you should also get an epub with a page-map.xml as well.

[Sabardeyn]

Kevin H.,
Not sure I understood the gist of this conversational thread.

Are you saying Amazon is now including the equivalent of a "page break" within the ebook text so that both digital and paper versions of a book have the exact same page content and numbering? Thus simplifying syncing citations between the two different media forms?

If the answer is yes, I imagine it's a fairly important step. It immediately opens ebooks up to more technical usage - like magazine articles, technical journals and textbooks. Where exactly locating and citing a source is mandatory.

[KevinH]

Hi,

Not quite ... page-breaks are just typically used to force text to start on a new page which is unreliable at best in a reflowable document.

Both epub and mobi/azw3 with APNX have the capability to encode information as to the start of each page in a specific hardcopy edition.

In epub, you use anchors/id= attributes to indicate the start of that page (not a pagebreak). Then in either a pagelist in the ncx or a separate page-map.xml file included in the epub, each of these anchors points is associated with its true page name (ie number or Roman numeral or ...). These are real page numbers with multiple schemes just as in the hardcopy version.

If an epub like that is passed into kindlegen, kindlegen will add a PAGE section to the mobi. This PAGE section becomes the APNX file when that book is delivered from Amazon. With the proper APNX file, a Mobi can show you on which page of the hardcopy book you are currently reading.

Calibre, does not yet understand these PAGE sections and the APNX file created by Calibre does not know or use multiple naming schemes such as Roman numerals and things like that. And many times these page start points are just chosen arbitrarily by Calibre.

So the most recent version of KindleUnpack (v072f) now understands these PAGE sections and will create a real APNX file for that mobi, and upon unpacking, will generate the proper page-map.xml entries.

I hope this is clearer. If not, just ask.

KevinH

Originally Posted by Sabardeyn

Kevin H.,
Not sure I understood the gist of this conversational thread.

Are you saying Amazon is now including the equivalent of a "page break" within the ebook text so that both digital and paper versions of a book have the exact same page content and numbering? Thus simplifying syncing citations between the two different media forms?

If the answer is yes, I imagine it's a fairly important step. It immediately opens ebooks up to more technical usage - like magazine articles, technical journals and textbooks. Where exactly locating and citing a source is mandatory.