KindleUnpack (MobiUnpack): Extracts text, images and metadata from Kindle/Mobi files

[AcidWeb]

As KindleGen decided to make my live miserable again I need a method to strip HD container from KF8 part - to reduce bloat when file is not distributed through Amazon ecosystem.

That would be nice optional feature of mobi_split class.

Source EPUB:
https://mega.nz/#!EtlkQY7b!3kPbl2SGc...Jfou7fwMO8jNeY
MOBI created with Kindlegen V2.9 build 1029-0897292 with -dont_append_source option:
https://mega.nz/#!80clTIZA!1Upz5luHY...WQO6qK93Up_WV0

After splitting it is easy to see that AZW3 part is much bigger. Images are embedded there two times. With different compression level.

Additionally I'm open to donate some money over PayPal to author of this feature. Price is negotiable.
Offer is no longer valid.

[KevinH]

FYI,

I am happy to accept patches into KindleUnpack master. Just make sure any changes run properly on both python 2.7 and python 3.4.

And whoever attempts this don't forget to change the metadata stored too. Including the metadata that says there is an HDcontainer in the first place and the extra BOUNDARY metadata, and the extra BOUNDARY section itself as well as the HDContainer and its sections

I strongly recommend looking at the output of DumpMobiHeader latest version to see the metadata and the extra sections to figure out what all has to go away.

Patches welcome.

KevinH

[AcidWeb]

Added sample files in my first post.

I should clarify: I'm placing code bounty on full-blown, proper Pull Request. So with KevinH blessing it can be merged with KindleUnpack codebase and be useful for the whole community.

Additionally: Can anybody confirm if PW3/Voyage use HD container at all? Last time I checked only Kindle Fire family used it.

Manga from Amazon Store is delivered to PW3 as almost empty AZW3 file and all images are in separate AZW6. I can only guess that that AZW6 file is created from HD container of original upload.

If that is true my goal described in my first post might be wrong. In this case HD container should be left alone and all other non-cover images embedded in KF8 should be replaced with Empty_Image/Resource_Placeholder.

Speculations... Only if I would have skills to check it myself :-P When somebody strip HQ container we will see if quality drop. That will answer if HQ container is used at all.

[pdurrant]

Originally Posted by AcidWeb

Speculations... Only if I would have skills to check it myself :-P When somebody strip HQ container we will see if quality drop. That will answer if HQ container is used at all.

If you download 'for transfer via USB' for a PW3, you should get a single .azw3 that's equivalent to the azw3/azw6 combination. If the file sizes are similar enough (to indicate that the images in the single azw3 and the azw3/azw6 are the same), you should be able to check with Kindle Unpack which images are getting sent to a PW3.

[AcidWeb]

Very interesting tip - thank you.

Delivered AZW3 have 0,1MB + 47MB AZW6
Downloaded file have 71,6MB.

EDIT:
I preformed extended autopsy.

File delivered wirelessly consists of a hollow AZW3 file and AZW6. I'm confirming KevinH research - AZW6 store images in JPEG XR format. Amazon really like to complicate simple things.

AZW3 downloaded from Amazon Store don't have HD container inside. All images are embedded normally in JPEG format.

Wireless delivery provide technically lower image quality than sideloaded version. Both use same resolution but compression level and bit depth is different.

But it is very hard to see any difference directly on the reader. JPEG XR + reduced bit depth simply provide much smaller files without any quality loss that matter.

HD Container embedded to KindleGen output still use normal JPEG format. EX magic apparently is done on Amazon side.

Sadly this entire corpse cutting not answered my main question: From where Kindle PW3/Voayge read images when we sideload bare KindleGen output with embedded HD container.

EDIT2:
Well. I found reason of bloat. I screwed up EPUB headers. book-type and original-resolution need to be meta name not meta property. I messed that up in last update. With them in place KindleGen don't create HD container at all.

My offer is not valid anymore.

[jemc]

I'm getting an odd little crash out of version 0.80. When I run kindleunpack.py from the command-line without redirecting output to a standard port or file this happens:

Spoiler:

C:\Temp\KU080>python lib\kindleunpack.py -h
KindleUnpack v0.80
Traceback (most recent call last):
File "lib\kindleunpack.py", line 1016, in <module>
sys.exit(main())
File "lib\kindleunpack.py", line 951, in main
print(" Based on initial mobipocket version Copyright © 2009 Charles M. H
annum <root@ihack.net>")
File "C:\Python27\lib\codecs.py", line 357, in write
data, consumed = self.encode(object, self.errors)
File "C:\Python27\lib\encodings\cp437.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\xa9' in position 4
9: character maps to <undefined>

The crash occurs regardless of command-line switches used.

When I redirect output: "python lib\kindleunpack.py -h >log.txt" kindleunpack.py executes correctly.

Version 0.77 works fine on my box in both the above cases.

I'm running Win8.1 64bit and tried both current ActivePython & Python.org v2.x & v3.x 32&64bit pythons with v0.80 - no joy.

No worries though, I've got my workaround for now.

[KevinH]

Hi,

That error happens in python 2.7 when the terminal encoding is not properly set to handle utf-8 characters. It is actually the copyright symbol in the "print" that is the issue.

If you are redirecting the output of the program in any way, python 2.7 is stupid enough to set sys.stdout to None. But that said, there is code in kindleunpack.py that should take care of this case for python 2.7.

if PY2:
    range = xrange
    # since will be printing unicode under python 2 need to protect
    # against sys.stdout.encoding being None stupidly forcing forcing ascii encoding
    if sys.stdout.encoding is None:
        sys.stdout = codecs.getwriter("utf-8")(sys.stdout)
    else:
        encoding = sys.stdout.encoding
        sys.stdout = codecs.getwriter(encoding)(sys.stdout)

If you have python 3.4 available, could you please test with it to see if it barfs about the same thing?

Thanks,

Kevin

Originally Posted by jemc

I'm getting an odd little crash out of version 0.80. When I run kindleunpack.py from the command-line without redirecting output to a standard port or file this happens:

Spoiler:

C:\Temp\KU080>python lib\kindleunpack.py -h
KindleUnpack v0.80
Traceback (most recent call last):
File "lib\kindleunpack.py", line 1016, in <module>
sys.exit(main())
File "lib\kindleunpack.py", line 951, in main
print(" Based on initial mobipocket version Copyright © 2009 Charles M. H
annum <root@ihack.net>")
File "C:\Python27\lib\codecs.py", line 357, in write
data, consumed = self.encode(object, self.errors)
File "C:\Python27\lib\encodings\cp437.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\xa9' in position 4
9: character maps to <undefined>

The crash occurs regardless of command-line switches used.

When I redirect output: "python lib\kindleunpack.py -h >log.txt" kindleunpack.py executes correctly.

Version 0.77 works fine on my box in both the above cases.

I'm running Win8.1 64bit and tried both current ActivePython & Python.org v2.x & v3.x 32&64bit pythons with v0.80 - no joy.

No worries though, I've got my workaround for now.

[jemc]

Originally Posted by KevinH

Hi,

That error happens in python 2.7 when the terminal encoding is not properly set to handle utf-8 characters. It is actually the copyright symbol in the "print" that is the issue.

If you are redirecting the output of the program in any way, python 2.7 is stupid enough to set sys.stdout to None. But that said, there is code in kindleunpack.py that should take care of this case for python 2.7.

<snip>

If you have python 3.4 available, could you please test with it to see if it barfs about the same thing?

Thanks,

Kevin

Ah, I've had other issues with the copyright symbol & utf-8...

Just tried it again with both Active Python 3.4.1 & Python.org 3.4.3 - crash.

Is it my understanding that you are saying this is 'supposed' to crash when I redirect output? --because the only way I can get v0.80 to properly execute from the command-line is to redirect output.

Best,
jemc

[KevinH]

Hi,
No, it is never supposed to crash! What is the error message and backtrace with Python 3.4?

Have you simply tried running via the GUI by running python on KindleUnpack.pyw. The gui spools the stdout to a scroll text pane.

Thanks,

KevinH

[jemc]

Originally Posted by KevinH

Hi,
No, it is never supposed to crash! What is the error message and backtrace with Python 3.4?

Have you simply tried running via the GUI by running python on KindleUnpack.pyw. The gui spools the stdout to a scroll text pane.

Thanks,

KevinH

Sorry, my dry sense of humor in the 'crash' remark didn't carry through.

Here is the error using Python.org v3.4.3:

Spoiler:

Microsoft Windows [Version 6.3.9600]
(c) 2013 Microsoft Corporation. All rights reserved.

C:\Temp>python lib\kindleunpack.py -h
KindleUnpack v0.80
Traceback (most recent call last):
File "lib\kindleunpack.py", line 1016, in <module>
sys.exit(main())
File "lib\kindleunpack.py", line 951, in main
print(" Based on initial mobipocket version Copyright \xa9 2009 Charles M.
Hannum <root@ihack.net>")
File "C:\Python34\lib\encodings\cp437.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_m ap)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\xa9' in position 49
: character maps to <undefined>

The gui version works fine via command-line startup as well as via mouse-click.

[KevinH]

Hi jemc,

To be honest, I am not sure how to best work around it. Your cp437 encoding just has no character that represents the copyright character and therefore any attempt to convert to your encoding from full unicode will fail. I have to look to see if the codec automatic output encoding setting allows for fallback replacement of some sort. I know if I convert each string manually in python, it does allow for fallback replacement but that kind of defeats the whole purpose of using the output codecs for sys.stdout

A better long term solution for you may be to get a shell/terminal program that supports the utf-8 encoding. This is standard on all Linux, and Mac OS X systems. There is code in Kindleunpack that enables the Windows cp equivalent to utf-8 but users must enable that encoding for it to work. I have heard that using the windows cp equivalent to utf-8 causes issues on some older Windows systems. I have no idea how well it works on newer Windows 8.1 or later systems.

Alternatively,you could always use the gui as it provides a full utf-8 terminal scroll pane window.

You mentioned a workaround of your own? What is it?

Thanks,

KevinH

[jemc]

Originally Posted by KevinH

Hi jemc,

To be honest, I am not sure how to best work around it. Your cp437 encoding just has no character that represents the copyright character and therefore any attempt to convert to your encoding from full unicode will fail. I have to look to see if the codec automatic output encoding setting allows for fallback replacement of some sort. I know if I convert each string manually in python, it does allow for fallback replacement but that kind of defeats the whole purpose of using the output codecs for sys.stdout

A better long term solution for you may be to get a shell/terminal program that supports the utf-8 encoding. This is standard on all Linux, and Mac OS X systems. There is code in Kindleunpack that enables the Windows cp equivalent to utf-8 but users must enable that encoding for it to work. I have heard that using the windows cp equivalent to utf-8 causes issues on some older Windows systems. I have no idea how well it works on newer Windows 8.1 or later systems.

Alternatively,you could always use the gui as it provides a full utf-8 terminal scroll pane window.

You mentioned a workaround of your own? What is it?

Thanks,

KevinH

Hi Kevin,

No problem - I didn't mean for you to spend so much time on this. Normally I use kindleunpack embedded inside another program I wrote that allows me to automate some text processing tasks, run kindlegen and finally run kindleunpack to split out the mobi8 file. I have each process dumping their output (stdout & stderr) to log files for later review. This still works fine. I ran across the above mentioned bug when I was testing some text processing changes I'd made while running kindlegen.py via command-line without redirecting the output. Anyway after checking everything on my end my first workaround was 'don't do that'. After you told me what was triggering the crash, I edited the code to change the copyright character to (C) in my copy of the code. Works fine now so I'm good. FYI: I've tripped over this problem over the years so tend to go 'kiss' with message text in direct command-line interfaces.

Again, thanks for your help!
Best,
jemc

[kovidgoyal]

@KevinH: FYI, the only way to robustly solve this is to use the win32 api. See the write_unicode_text() function in the calibre source code in utils/terminal.py

[KevinH]

Hi Kovid,

Thank you. I will do that.

KevinH

Originally Posted by kovidgoyal

@KevinH: FYI, the only way to robustly solve this is to use the win32 api. See the write_unicode_text() function in the calibre source code in utils/terminal.py

[mattmc]

Sorry, this may not be particularly contributive to the discussion, but I had a look at the KindleUnpack source and I just had to say--wow, quite an engineering effort! ~5k lines of code, excluding comments. An excellent job, for sure, and thank you.