KindleUnpack (MobiUnpack): Extracts text, images and metadata from Kindle/Mobi files

[elmimmo]

I tried to unpack a deDRM'ed AZW file but got the quote below. Any hint how to make it work with this AZW file?

MobiUnpack 0.53
   Based on initial version Copyright © 2009 Charles M. Hannum <root@ihack.net>
   Extensions / Improvements Copyright © 2009-2012 P. Durrant, K. Hendricks, S. Siebert, fandrieu, DiapDealer, nickredding.
   This program is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation, version 3.
Unpacking Book...
Palm DB type:  BOOKMOBI
Mobi Version:  6
Codec:  utf-8
Title:  ••••••••••
Palmdoc compression

Processing Mobi format Ebook ...
Warning: Unknown metadata with id 405 found
Warning: Unknown metadata with id 406 found
Warning: Unknown metadata with id 407 found
Warning: Unknown metadata with id 403 found
Unpack raw markup language
Unpacking images, resources, fonts, etc
Traceback (most recent call last):
  File "/Users/jorge/bin/mobi_unpack.py", line 1087, in <module>
    sys.exit(main())
  File "/Users/jorge/bin/mobi_unpack.py", line 1075, in main
    unpackBook(infile, outdir)
  File "/Users/jorge/bin/mobi_unpack.py", line 957, in unpackBook
    process_all_mobi_headers(files, sect, mhlst, K8Boundary, False)
  File "/Users/jorge/bin/mobi_unpack.py", line 705, in process_all_mobi_headers
    usize, fflags, dstart, xor_len, xor_start = struct.unpack_from('>LLLLL',data,4)
struct.error: unpack_from requires a buffer of at least 20 bytes

[pdurrant]

Use version 0.54, which has some code to stop corrupt fonts crashing the procedure.

[DiapDealer]

Definitely a funky embedded font situation. Sure would like to get a hold of a legal, non-DRMed KF8 file that exhibits this behavior to see exactly what's going on.

To my knowledge... v0.54 should preserve the entire raw resource record as a *.failed file in the unpacked data. That was done in the hopes that someday, someone would be able to take a look at it and see what might be going on. I'd settle for the file-size of the .failed file, for starters, if anyone comes across the situation.

[elmimmo]

Originally Posted by DiapDealer

v0.54 should preserve the entire raw resource record as a *.failed file in the unpacked data

v0.54 worked, and it did exactly as you described: "font00002.failed, font00003.failed…". Images and HTML was extracted correctly. Thanks.

[DiapDealer]

Did you happen to catch the sizes of any of those .failed files?

[pdurrant]

Updated to v0.55, incorporating changes to fix the double startoffset metadata in the extracted .azw3 file.

Also the AppleScript has a spiffy new icon.

[amarie0]

Awesome artwork Paul! https://twitter.com/amarie/status/261829113550082049

[DiapDealer]

Some proposed changes to MobiUnpack for perusal and opinions. Very minor.

Changes to mobi_opf.py: escaping even more metadata so invalid OPFs are avoided. I honestly thought I had the bases covered when I escaped the contents of the two "handler" methods, but metadata like "Updated Title" was still falling through the cracks when containing '&' and such. I escaped "Subject" tag contents for good measure as well.

Changes to mobi_unpack.py: this one's frankly because there's a bug in my plugin that I flat-out can't fix without a change to the underlying MobiUnpack code. I feel a little sleazy requesting the change, but in all honesty, the bug could affect any programs that may want to import some of Mobi_Unpack's code as modules in the future (or at least that's what I'm telling myself).

Anyway... python's unicode function will blow up if you try to use it on a string that's already unicode. And since calibre's strings are unicode by default ... passing a unicode filepath string to MobiUnpack's unpackBook method will blow up later when an attempt to "re"-unicode the basename portion of that string is made. I've added a simple check to determine whether or not the string is already unicode before further processing. Without this change, no unicode filepath strings can be passed to the main method. I have to convert them to normal strings first -- which of course, means that no non-ascii characters can be a part of the file's pathname. Unless I'm missing something obvious, which is always quite probable.

I also have a weird test-case azw3 that brings up a fantastically convoluted encoding issue, but I may need to ponder that a bit more yet.

EDIT: to those who downloaded the very first zip, I apologize, I made a mistake and tried to upload the correct one as soon as possible, but a couple of people beat me. Download again for the latest—and what I think is the correct—version.

[pdurrant]

Originally Posted by DiapDealer

Some proposed changes to MobiUnpack for perusal and opinions. Very minor.

They sounds sensible. I'll take a look tomorrow.

I don't see any harm in changing Mobiunpack to be a bit more robust when it comes to unicode. It might make sense to be a bit more thorough, thanks.

[pdurrant]

MobiUnpack 0.56 now uploaded to first post.

The AppleScript has been enhanced to dump debug info if option key held down on launch.

[DiapDealer]

The first post still seems to have v0.55 attached to it.

[pdurrant]

Well, how did I manage that?

Now updated with version 0.57. I found a problem with splitting a file downloaded from the KDP website preview, now fixed.

[DiapDealer]

I'll see your version 0.57 and raise you a version 0.58 that outputs the original kindlegen build log -- which is apparently all the new 'CMET' record is comprised of that kindlegen 2.7 includes in its output.

[pdurrant]

Originally Posted by DiapDealer

I'll see your version 0.57 and raise you a version 0.58 that outputs the original kindlegen build log -- which is apparently all the new 'CMET' record is comprised of that kindlegen 2.7 includes in its output.

Now uploaded to the first post. Nice addition.

[steppe]

Originally Posted by adamselene

Most of this post now by pdurrant.

Mobiunpack is a python script that takes a Kindle/Mobipocket ebook and extracts the HTML, images and metadata contained in the ebook, and puts them in a form suitable for passing to KindleGen.

Thanks guys for this gadget and for the updates. It is invaluable when working with picture books.