KindleUnpack (MobiUnpack): Extracts text, images and metadata from Kindle/Mobi files

[KevinH]

Hi AcidWeb,

Originally Posted by AcidWeb

Well... I consider many coding standards used here as messy but this is not my project so I will just shut up and and focus on making this code work on both Python versions.

The core already does work on both. The only pieces remaining are the gui front-end KindleUnpack.pyw and its support code: scrolltextwidget, askfolder, and prefs.

I have already started working on that as well.

Edit:
Not bad. CLI version look good.
I presume that not converted prints was caused by fact that you not touched GUI part yet.

Most probably, see above.

I hate Tk so no help from me there. Sorry.

No problems, I have done the core code, doing the rest should not be a big problem.

One small trap: unescape function used in mobi_opf.py will be removed in Python 3.5.

Thanks for the head's up. I see I can now use:

import html
html.unescape()

in its place so I will simply do that as the compatibility_utils already handles most of that and can be easily modified to do that as well.

Take care,

KevinH

[Peter Ahlstrom]

I'm getting a bug in version 75 of the script (OS X version) today, but only on some files. Here's the report:

Traceback (most recent call last):
File "/Applications/KindleUnpack v0.75.app/Contents/Resources/kindleunpack.py", line 968, in <module>
sys.exit(main())
File "/Applications/KindleUnpack v0.75.app/Contents/Resources/kindleunpack.py", line 957, in main
unpackBook(infile, outdir, apnxfile, epubver, use_hd)
File "/Applications/KindleUnpack v0.75.app/Contents/Resources/kindleunpack.py", line 871, in unpackBook
process_all_mobi_headers(files, apnxfile, sect, mhlst, K8Boundary, False, epubver, use_hd)
File "/Applications/KindleUnpack v0.75.app/Contents/Resources/kindleunpack.py", line 761, in process_all_mobi_headers
imgnames, image_ptr = processCRES(i, files, imgnames, sect, data, beg, image_ptr, use_hd)
File "/Applications/KindleUnpack v0.75.app/Contents/Resources/kindleunpack.py", line 300, in processCRES
outimg = os.path.join(imgdest, imgname)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 75, in join
if b.startswith('/'):
AttributeError: 'NoneType' object has no attribute 'starts with'

Judging by that, this may actually be some kind of python 2.7 bug, but I'm not sure... (And none of my images start with "/")

This is on a mobi today that I made by uploading an epub to KDP and downloading the preview file. Doing that and extracting via KindleUnpack v 75 was working earlier today, but some of the html and css coding has changed since then. The current file will unpack using v 64 of the script, but that's of course lacking features like the HD Image stuff. If I run the same epub through kindlegen, the resulting mobi will unpack fine with v 75.

Earlier today I could upload an ePub with image files larger than 127K and KDP would spit out a .mobi with the large files in it—but I had font embedding problems. I've fixed the font embedding problems, but now I can't tell if the large jpegs are still in there, since KindleUnpack is failing.

[KevinH]

Hi Peter,

Yes somethign is broken but I am not sure what. The error you are getting means that you are trying to unpack some HDImage and either its name is missing somehow or its imgdest (the path to put the image in) is missing.

If this happens only with KDP Preview, my guess is they have somehow changed things and are only providing HD images which Kindlegen does not.

So here is a workaround just to check. Please try *not* adding the -i switch that turns on using HDImages to replace lower res versions. They will still be unpacked and stored right in the HDImages folder so you won't lose anything.

See if that lets the unpack succeed. If so, KDP may be doing something with CRES sections and High Res images we don't know anything about yet.

Either way, if you can provide a copy of the book to me, I should be able to track down what is going on and get a fixed version to you to test with.

If you want it to be private, simply PM me here with a url to grab it from and I will throw it out once I have figured out why this is happening and how to deal with it without breaking Kindlegen generated files either.

Alternatively, you could try running DumpMobiHeader_v018.py on the book and it may tell us what new sneakiness they are up to?

Place DumpMobiHeader_v018.py on your Desktop right beside the KDP preview. Then open Terminal.app and type the following:

cd
cd Desktop
python ./DumpMobiHeader_v018.py BOOK_FILE_NAME_HERE > header.txt
exit

Then send to me or post the header.txt file and it may tell us something more.

Take care,

KevinH

Originally Posted by Peter Ahlstrom

I'm getting a bug in version 75 of the script (OS X version) today, but only on some files. Here's the report:


Judging by that, this may actually be some kind of python 2.7 bug, but I'm not sure... (And none of my images start with "/")

This is on a mobi today that I made by uploading an epub to KDP and downloading the preview file. Doing that and extracting via KindleUnpack v 75 was working earlier today, but some of the html and css coding has changed since then. The current file will unpack using v 64 of the script, but that's of course lacking features like the HD Image stuff. If I run the same epub through kindlegen, the resulting mobi will unpack fine with v 75.

Earlier today I could upload an ePub with image files larger than 127K and KDP would spit out a .mobi with the large files in it—but I had font embedding problems. I've fixed the font embedding problems, but now I can't tell if the large jpegs are still in there, since KindleUnpack is failing.

[KevinH]

Hi All,

So given the setup.cfg setting that Kovid so nicely supplied, and his nice autopep8 script, and lots of file editing, the core python code (inside lib/) should now pass flake8 tests on both master and python2and3 branches.

The other nice thing to see was there were actually no functionality bugs that could impact anything other than debug output!

Thanks to everyone who helped get this thing underway.

Thanks,

KevinH

[Peter Ahlstrom]

Kevin, thank you. I have sent you a link to the file. I edited the AppleScript to not add the -i flag, and everything extracted correctly.

[KevinH]

Hi Peter,

Thanks, I will grab it. Since removing the -i worked then the problem really is HDImage related. Please look in the unpack folder in the HDImages folder to verify the right HD Images were unpacked. If you are going to run the unpacked ebook through kindlegen, don't forget to manually copy those images and rename them to replace the corresponding image in the mobi8/OEBPS/Images/ folder (which is what the -i option would have done for you.)

I'll try to track down a fix for you today or tomorrow at the latest.

Thanks,

KevinH

Originally Posted by Peter Ahlstrom

Kevin, thank you. I have sent you a link to the file. I edited the AppleScript to not add the -i flag, and everything extracted correctly.

[Peter Ahlstrom]

Thanks, Kevin. The correct HD images were indeed unpacked.

Is there some advantage to running an unpacked mobi through kindlegen again (if I still have all the original source files)? I used to do this a few years ago in the only-mobi7 days in order to simplify the html that kindlegen spat out, but there doesn't seem to be an effective way to do this nowadays (to edit the html directly and have it affect only the mobi7 portion—and I'm not talking about media queries).

Essentially, nowadays I run KindleUnpack only to see exactly what will be delivered to Kindle users.

[KevinH]

Hi Peter,

There is no need to pass it through kindlegen again after unpacking unless you want to make quick changes based on what was unpacked versus the original source.

The reason you saw a difference was that your fonts were actually included properly and we had never tested a case with both HD Image unpacking and embedded Fonts!

So the problem was entirely ours. Since the Fonts are stored before the Images, we were trying to unpack the HDImages over the lower res versions but grabbed the wrong image name since the addition of fonts changed our count.

With the new changes in place, you can safely use the -i switch to unpack things. The fix touched a number of files because it changed our thinking about what things actually mean inside the .mobi. I will post a KindleUnpack_v077.zip version shortly that should take care of things.

Thank you so much for submitting your bug report and test case as it helps us to better understand the inner workings of the mobi format!

Take care,

KevinH

[KevinH]

Hi All,

Attached is a KindleUnpack_v077 bug fix release:

Changes included since v075:

- clean up to pass adjusted flake8 style issues

- bug fix to rename mobi7 output to book.html, toc.ncx, content.opf to mirror what is done in mobi8 and to prevent issues in creating a valid opf

- bug fix for unpacking HDImages when embedded Fonts are included

- other small changes and bug fixes

Please give it a try and post any bugs here.

Also all changes have already been committed to both branches of the KindleUnpack github repository:

https://github.com/kevinhendricks/KindleUnpack

KevinH

[KevinH]

Hi All,

Just a head's up to say the port to both Python 2 and 3 is now complete. If you are looking for something to test, please checkout the python2and3 branch from KindleUnpack's github repository.

After a sufficiently long testing period, this version will be merged into and become the master and then released as KindleUnpack_v080. So finding and reporting any bugs now would be welcome!!!!

Tested with Python 2.7.7 and Python 3.4.1 on Mac OS X 10.9.5

Thanks,

KevinH

[eschwartz]

@KevinH,

You are configured with an email address "kevin.b.hendricks@noreply-github.com" but the appropriate GitHub email address is "kevinhendricks@users.noreply.github.com" (which you actually did use, once).

[KevinH]

Hi eschwarts,

Does this actually matter for anything important or functional? Neither are real e-mail addresses as far as I can tell.

Thanks,

KevinH

Originally Posted by eschwartz

@KevinH,

You are configured with an email address "kevin.b.hendricks@noreply-github.com" but the appropriate GitHub email address is "kevinhendricks@users.noreply.github.com" (which you actually did use, once).

[eureka]

Originally Posted by KevinH

Does this actually matter for anything important or functional? Neither are real e-mail addresses as far as I can tell.

One of them is recognized by GitHub web-interface as e-mail of your GitHub account and author/committer information with that e-mail is represented as a functional link. Look at commit 0d0e21f.

BTW, I guess link to GitHub-provided releases archive in new version announcement would be useful (for diversification of downloads if not for backup purposes).

[KevinH]

Hi eureka,
Yes that would be a good idea but I had not figured out how to push the historic release tags from my local tree to github yet, so there were no releases to link to when I made the post. I only found out how to do that today.

In the futue, I will post that link as well, but the primary download will always be here as this is the community we serve.

Take care,

KevinH

[eschwartz]

Originally Posted by eureka

One of them is recognized by GitHub web-interface as e-mail of your GitHub account and author/committer information with that e-mail is represented as a functional link. Look at commit 0d0e21f.

BTW, I guess link to GitHub-provided releases archive in new version announcement would be useful (for diversification of downloads if not for backup purposes).

Yep, basically you get a pretty icon and a clicky to your GitHub profile in the commit log plus credit here: https://github.com/kevinhendricks/Ki...kevinhendricks

No functional difference. But if you didn't want to have the clicky you could always have used a null email rather than a fake one.