KindleUnpack (MobiUnpack): Extracts text, images and metadata from Kindle/Mobi files

[tkeo]

Hi Kevin,

Originally Posted by KevinH

Hi tkeo,
Once we get enough "its working" or by lack of "bug reports" please release it here as an official KindleUnpack_v073 version!

Is it OK to include get_image_size() function for next release?

And have you made GUI for epub3 support? If so, please upload it, I'd like to merge it.

Thanks,

[elmimmo]

In KF8 FXL comics, Kindlegen has always compressed images heavier than 800KB to something below that, but Amazon purportedly reports, according to the blog post Kindle Image File Size Increase Confirmed, that

From Version 2.9 of Kindlegen, images up to 5MB are retained for High Definition devices.

According to that blog’s author, that is purportedly the case even when using Kindlegen "-dont_append_source" Option:

The kindlegensrc.zip archive is not required in order for the HD images to be delivered, as these are stored in a new HD Container section of the converted mobi file.

The person added via Twitter that:

This info came via email communication with Amazon.

Any chance that KindleUnpack will be updated to extract the contents of this HD Container when kindlegensrc.zip is not available?

[KevinH]

Hi Testers,

Here is the very latest experimental version of KindleUnpack that should be able to generate APNX files if the mobi passed in has a PAGE section.

It also has lots of GUI improvements from DiapDealer and experimental EPUB3 support from tkeo.

Please let us know what if any problems you run into with any of these new features in this experimental release.

Thanks,

KevinH

Originally Posted by KevinH

Hi All,

I have a new experimental version of KindleUnpack that has all of tkeo's latest changes for epub3 support and that will generate an apnx file if the mobi passed in has a PAGE section.

So if have an epub that has either a pagelist in the ncx or a page-map.xml, and you pass it into kindlegen, kindlegen will create a mobi with a PAGE section.

If you then pass that mobi into KindleUnpack, and split it, you should now find a:
- mobi7-BOOKFILENAME.mobi
- mobi7-BOOKFILENAME.apnx
- mobi8-BOOKFILENAME.azw3
- mobi8-BOOKFILENAME.apnx

all in the unpacked output directory.

I don't have access to an actual Kindle, but I am hoping if you side load either version and its associated apnx, you should see actual page numbers that exactly match what what was provided in the original epub.

So if anyone has an actual Kindle they can test with and an epub with real page numbers in either the pagelist in the ncx or a separate page-map.xml file, I would love to have help debugging the apnx generation and getting it to actually work.

Please let me know if you are willing to help and I will upload the experimental version of KindleUnpack for you to try.

Thanks,

KevinH


PS: FWIW, it works perfectly with my Kindle.app on my Mac OS X machine. Page numbers show up just fine when I move the .azw3 and its .apnx into "My Kindle Content" folder.

So I would love to know if this works for standalone Kindles and other Kindle apps.

[KevinH]

Hi tkeo,

Originally Posted by tkeo

Is it OK to include get_image_size() function for next release?

Okay, that is a popular piece of code. I could not find the original site "draco.boskant.nl" anywhere online. Nor did a Google search of draco and boskant and nl turn up anything really. I did find an src.rpm from 2006 linked to by FileWatcher AltLinux. And then by searching altlinux repositories I finally found the Draco-0.99.4.tar.gz and downloaded it.

And yes the routine in question came from image.py and was released under GPL version 2.

Also, the Internet Archive WayBack Machine (web.archive.org) did archive the main page from http://draco.boskant.nl/ and it confirmed it was released under the GPL version 2 back in 2003 I believe.

Also I did the following Google search:

"Determine the image type of fhandle" and draco

And I found that exact same code in at least 3 places all on github.

- in Aveenstra/Images_To_ePub/_ePubMaker.py under the GPL 3 license

- in evfredericksen/oodocx/oodocx/helper_functions.py under a BSD License

- in SublimeText3-Packages/SideBarEnhancements/SideBar.py under a GPL (no version specified) license

I also found it submitted to the Django source-code repository.

So keeping it in our tree under the GPL 3 license is just fine given its use under the exact same licence or free-er.

Originally Posted by tkeo

And have you made GUI for epub3 support? If so, please upload it, I'd like to merge it.

I have an experimental version that uses an option menu that defaults to epub2. It is still being debugged. When I built it I broke some of DiapDealer's new features, and I want to get that fixed before posting it. Also, I think DiapDealer will want to handle the setting of epub version in his new ini file so I will ask him for help on that.

Please note: I have made a number of minor typo fixes and other changes in your v072c (spelling of HIEGHT -> HEIGHT, etc) and added support for converting PAGE sections to APNX files (and DiapDealer has done a lot of work on the TkGUI) so please use the KindleUnpack_v072d_test I just posted as the basis of any new feature work so that all of these improvements are not lost.

Thanks,

KevinH

[KevinH]

Hi,

Any of the very recent experimental releases already does that. The HDImages are put in their own HDImages directory when unpacked.

KevinH

Originally Posted by elmimmo

In KF8 FXL comics, Kindlegen has always compressed images heavier than 800KB to something below that, but Amazon purportedly reports, according to the blog post Kindle Image File Size Increase Confirmed, that



According to that blog’s author, that is purportedly the case even when using Kindlegen "-dont_append_source" Option:



The person added via Twitter that:



Any chance that KindleUnpack will be updated to extract the contents of this HD Container when kindlegensrc.zip is not available?

[Doitsu]

Originally Posted by KevinH

Here is the very latest experimental version of KindleUnpack that should be able to generate APNX files if the mobi passed in has a PAGE section.

I did a quick test with both pagemap and pagelist based page numbers. The generated .apnx files worked on my K3, but after a certain point the page numbers were off by minus one (see screenshot).
However, this might have been caused by incorrect pagemap or pagelist definitions.
The .apnx files didn't work on my PW2 though. When I put them in the .sdr (sidecar) folder they were deleted and when I put them in the same folder as the .azw3 files they were apparently ignored.

It looks like Amazon has modified the .apnx format for PW Kindles because the fake page numbers generated by Calibre don't work either.

Please find attached the test files that I used and the sidecar files generated by the PW2 firmware.

EDIT: The tool works fine. For details see my next post.

[NiLuJe]

There's an extended header (a few more fields needed, IIRC) needed for KF8 files. Calibre should be using it when processing KF8 files, though.

I'll take a look.

@Doitsu: Which FW version are you running, and has your device ever shown 'page numbers' since the last factory reset? (be it in an M7 or KF8 file).
IIRC, there's a weird quirk when it sometime doesn't like 'custom' APNX files until something happens (AFAICT, either getting a proper KF8+page book OTA, or opening an M7 one).

Because Calibre's APNX work for me, and KindleUnpack appears to be doing the right thing in regards to the extended header for KF8 APNX.


I'm not sure I have a properly setup ePub book on hand to try it from scratch, and I'm a bit short on time until next week, but if someone has a dummy reduced testcase that I can try, I'd be happy to . [Barring that, I'll try with the files Doitsu generated].

[KevinH]

Hi Doitsu,

From the looks of what you sent, yes your PW2 does NOT use or read apnx files. Instead it downloads into sidecar two files. The first is the .azwf which includes history and timing info (almost like the .mbp file). The second is an .azw3r that keeps the apnx data but converted into some other strange format.

It almost looks like the .azw3r file is a json or other serialized object version of both an annotation.cache.object, and an apnx.key. The apnx.key has the exact same pagemap information except for completely out of order and with some extra pieces I am not sure about. It may in fact be a serialized java object, a serialized json object, or some other object serializer used for storage and retrieveal in a binary file.

So it looks like creating an apnx for a PW2 is not going to be helpful. Sorry about that. But Amazon seems to keep going out of its own way to create a support nightmare for itself with all of its different machine specific formats.

I will look into the off by one on the KW and see if I can track down why that is happening. Hopefully I can add a fix for that.

Thanks for testing things and providing the sidecar files which helped me track down where the apnx was being stored on the PW2.

[KevinH]

Hi Doitsu,

I am trying to track down the "off by one" issue but I am not having any luck.

Here is your original epub pagemap info

<page-map xmlns="http://www.idpf.org/2007/opf">
    <page href="TOC.xhtml" name=""/>
    <page href="chapter1.xhtml#id1" name="i"/>
    <page href="chapter1.xhtml#id2" name="ii"/>
    <page href="chapter1.xhtml#id3" name="iii"/>
    <page href="chapter2.xhtml#id4" name="4"/>
    <page href="chapter2.xhtml#id5" name="5"/>
    <page href="chapter2.xhtml#id6" name="6"/>
    <page href="chapter3.xhtml#id7" name="7"/>
    <page href="chapter3.xhtml#id8" name="8"/>
    <page href="chapter3.xhtml#id9" name="9"/>

Here is what the created azw3 PAGE section says about page names and offsets

[None, 'i', 'ii', 'iii', '4', '5', '6', '7', '8', '9']
[319, 1204, 4739, 8106, 12062, 15606, 18982, 22948, 26492, 29868]


Here is what the generated apnx file from that page section says about page numbers and offsets :

Decode APNX
mobi8-mobi8-pagemap.apnx .APNX
1 75 10 32
2 r 1
5 a 4
[None, 'i', 'ii', 'iii', '4', '5', '6', '7', '8', '9']
[319, 1204, 4739, 8106, 12062, 15606, 18982, 22948, 26492, 29868]


So the page names and page offsets are absolutely identical as they should be.

And here is the page-map.xml generated when the azw3 was unpacked by KindleUnpack

<page-map xmlns="http://www.idpf.org/2007/opf">
<page name="i" href="Text/part0001.xhtml#id1" />
<page name="ii" href="Text/part0001.xhtml#id2" />
<page name="iii" href="Text/part0001.xhtml#id3" />
<page name="4" href="Text/part0002.xhtml#id4" />
<page name="5" href="Text/part0002.xhtml#id5" />
<page name="6" href="Text/part0002.xhtml#id6" />
<page name="7" href="Text/part0003.xhtml#id7" />
<page name="8" href="Text/part0003.xhtml#id8" />
<page name="9" href="Text/part0003.xhtml#id9" />

The only difference I can see is that the unnamed page at the beginning of the document that represents the TOC.xhtml did not get assigned a name in the unpacked page-map because that page had no name (it was a nullstring). But that is not the cause of the issue as all of the id="idX" tags align perfectly with the correct page names.

Hmm ... so nothing appears to be off by one here.

So can you locate where things appearing to be off by one starts and let me know so that I can try and see if the original xhtml coding is an issue or if the issue is just that more than one page is on the screen at the same time (which is device specific).


Thanks,

KevinH

[Doitsu]

Originally Posted by KevinH

So it looks like creating an apnx for a PW2 is not going to be helpful. Sorry about that.

No problem. I'm glad that you looked into the possibility of creating such a tool at all when hardly anybody else did. And your tool does seem to work with other KF8-capable Kindles and apps, which is a major improvement over the "fake page numbers" that Calibre generates.

I really don't understand why Amazon don't release the tool that they use to generate .apnx and .azw3r files.

In case Kovid read this, my model's serial number prefix is 90D4 and my firmware is 5.4.3.2. The latest Calibre version obviously cannot generate .azw3r files, because the format hasn't been reverse engineered yet, and transferring .apnx files to sidecar folders is pointless, because they'll be deleted when the book is opened for the first time.

BTW, I'm still puzzled about the "page number drift" that happened on my K3 after the first three pages. I most likely specified the page targets incorrectly or overlooked some other constraints. Maybe someone else who successfully implemented page numbers for a commercial Kindle book can shed some light on this.

Originally Posted by KevinH

So can you locate where things appearing to be off by one starts and let me know so that I can try and see if the original xhtml coding is an issue or if the issue is just that more than one page is on the screen at the same time (which is device specific).

Each chapter should start with an odd page number, you can see this for example, when you open chapter 3, which should start with page 7, but page 6 is being displayed. All .html pages are identical, except for the page target ids.

[KevinH]

Attached is an updated test version of KindleUnpack_v072f_test.zip.

New features:

- Greatly Improved GUI with support for:

- epub version setting
- using HDImages if present
- full saving and retrieving of your GUI settings as preferences

All Thanks goes to DiapDealer for the above!

- HD_Images will now automatically overwrite their lower res cousins if that option is selected.

- PAGE Unpacking has minor bug fixes

- bug fixes for Windows machines using paths that require full unicode.

All features should now be implemented and useable.


Testers and Interested Users:

Please give this a try and report back any issues

Co-Developers:

Please use this version as the basis for any future work. Many files have been touched and bugs with paths fixed.

Hopefully we can now just bug fix and polish before an official release. I still would like to prune out any redundant code. For example I have now imported get_image_type from mobi_cover to help remove redundant code in processImage and processCRES in kindlenpack.py and fixed a few bugs related to that. I would also like to greatly reduce the redundancy in the opf creation code as well before going final.

[NiLuJe]

Okay, gave a go at Doitsu's pagemap file, and, yup, it definitely works for me.

Even the roman numerals stuff works (which I'd never actually seen before, so, yay).

On most devices supporting KF8 (AFAICT, everything *except* the K3, though I haven't checked the K3 thing), the apnx file *HAS* to go in the sidecar (.sdr) folder though.
Since I'm doing this over SSH, I pushed the files in reverse to make sure the scanner didn't do something stupid (ie. pushed the sidecar folder, the apnx, and then the azw3).

And yes, the framework consumes the APNX file and builds the azw3f & azw3r couple.

(Yes, it's a Touch, but it's running on the 5.3 branch, results should be identical on a PW/PW2).

FWIW, I'm seeing Page 6 after a goto Chapter 3, too. Could it be because the id anchor (or however that thing works ) comes a tiny bit too late? No idea how the reader parses stuff, but there might be some tricky caching or stupid behavior involved.

I'm also not seeing a page number for the "Chapter 1" entry in the GoTo menu, I'm guessing for the same kind of reasons that the 'off-by-one' thing.

If I "GoTo" page 7, I don't see the chapter heading anymore, (ie. ==== PAGE 7 ==== is at the top), and I'm indeed on page 7. Haven't looked at the source, but that would seem to correlate with my hunch.

[KevinH]

Originally Posted by Doitsu

BTW, I'm still puzzled about the "page number drift" that happened on my K3 after the first three pages. I most likely specified the page targets incorrectly or overlooked some other constraints. Maybe someone else who successfully implemented page numbers for a commercial Kindle book can shed some light on this.

Each chapter should start with an odd page number, you can see this for example, when you open chapter 3, which should start with page 7, but page 6 is being displayed. All .html pages are identical, except for the page target ids.

Okay but the epub code puts the page anchors after the chapter heading so the chapter heading is actually on the previous page (technically). So if they show you the page of the first character on the page, that is in fact correct.

So could you please try editing your epub to change things slightly for the start of every chapter. Try going from your current code where the actual page start location actually comes after the chapter heading to before it as follows:

from:

<body>
  <h3>Chapter 1</h3>
  <p><a id="id1"></a><b>==== PAGE 1 ====</b>

to

<body id="id1">
  <h3>Chapter 1</h3>
  <p><b>==== PAGE 1 ====</b>

or to

<body>
  <h3 id="id1">Chapter 1</h3>
  <p><b>==== PAGE 1 ====</b>

and let me know if once it gets converted by kindlegen and the apnx is split out if it will show up properly. If so, I think the off by one is simply a coding issue. If not, then we have a bug someplace.

Thanks,

Kevin

[NiLuJe]

@KevinH: Yay! That confirms my hunch (cf. my previous message, which has been edited 7 billion times already ^^) .

I'll be happy to check updated files on my end too .

[KevinH]

Hi NiLuJe,

Yes, please try both alternative epub changes for each chapter and see if either will work properly and let me know what you see. I really don't like using id= inside a "body" tag as the target should then just be the top of file for consistentcy and simplicity.

I am glad to hear that a sideloaded apnx file can be consumed and that the .azw3r and .azw3f funny pieces are created during reading. I hope Doitsu can do the same thing with his PW2.

One other variation to try might be:

<body>
  <a id="id1"></a>
  <h3>Chapter 1</h3>
  <p><b>==== PAGE 1 ====</b>

This would make it clearest that the anchor starts before the chapter heading.

Thanks!

KevinH

Originally Posted by NiLuJe

@KevinH: Yay! That confirms my hunch (cf. my previous message, which has been edited 7 billion times already ^^) .

I'll be happy to check updated files on my end too .