KindleUnpack (MobiUnpack): Extracts text, images and metadata from Kindle/Mobi files

[KevinH]

Hi tkeo,

Okay I modified things to not change the stdout.encoding on python 2 unless it was set to None (which can happen when you pipe or redirect to a file). I also ran everything through reindent.py that will fix all indentations to be exactly 4 chars, remove all tabs, and strip all whitespace from the end of each line.

This way your IDE should not be seeing so many things changed when you post your next patch.

I have attached the updated and hopefully closer to final version of nlib.zip here. I will remove all older broken versions.

Thanks,

KevinH

[tkeo]

Hi Kevin,

1. in compatibility_utils.py, under Windows, you can not use the following code, as it will break almost on Windows machines that do not use the cp you use.

+
+ # Conversion of argv to unicode without using cdll.kernel32.GetCommandLineW
+ FILE_SYSTEM_ENCODING = sys.getfilesystemencoding()
+ uargv = []
+ in_double_quotations = False
+ for arg in sys.argv:
+ if not isinstance(arg, unicode):
+ arg = arg.decode(FILE_SYSTEM_ENCODING)
+ if in_double_quotations:
+ if arg[-1] == u'"':
+ in_double_quotations = False
+ arg = arg[:-1]
+ uargv[-1] = uargv[-1] + u' ' + arg
+ else:
+ if arg[0] == u'"':
+ arg = arg[1:]
+ if arg[-1] == u'"':
+ arg = arg[:-1]
+ else:
+ in_double_quotations = True
+ uargv.append(arg)
+ return uargv
+

It is much better to use the kernel32 call, and kindleunpack.py has always used this routine (see utf8_utils.py for utf8_argv) in one form or another. It is needed for full unicode (at least utf-16) compliance on all Windows machines under python 2.

This is for debug purpose code.
Since in my IDE (spyder), parameters are not able to passed by kernel32 call.
Could you keep it in until the end of this debug?

Thanks,

[tkeo]

Hi Kevin,

These are errors of the latest nlb code.

1. test1.azw3 with python 3 (fixed):

Spoiler:

Unpacking raw markup language
Traceback (most recent call last):
File "kindleunpack.py", line 1016, in <module>
sys.exit(main())
File "kindleunpack.py", line 1004, in main
unpackBook(infile, outdir, apnxfile, epubver, use_hd)
File "kindleunpack.py", line 918, in unpackBook
process_all_mobi_headers(files, apnxfile, sect, mhlst, K8Boundary, False, ep
ubver, use_hd)
File "kindleunpack.py", line 835, in process_all_mobi_headers
processMobi8(mh, metadata, sect, files, imgnames, pagemapproc, k8resc, obfus
cate_data, apnxfile, epubver)
File "kindleunpack.py", line 493, in processMobi8
linktgt += '#' + idtext
TypeError: Can't convert 'bytes' object to str implicitly

2. test1.azw3 with python 3:

Spoiler:

Traceback (most recent call last):
File "kindleunpack.py", line 1016, in <module>
sys.exit(main())
File "kindleunpack.py", line 1004, in main
unpackBook(infile, outdir, apnxfile, epubver, use_hd)
File "kindleunpack.py", line 918, in unpackBook
process_all_mobi_headers(files, apnxfile, sect, mhlst, K8Boundary, False, ep
ubver, use_hd)
File "kindleunpack.py", line 835, in process_all_mobi_headers
processMobi8(mh, metadata, sect, files, imgnames, pagemapproc, k8resc, obfus
cate_data, apnxfile, epubver)
File "kindleunpack.py", line 531, in processMobi8
usedmap = htmlproc.buildXHTML()
File "mobi_html.py", line 337, in bu
ildXHTML
replacement = b'"../' + utf8_str(pdir) + b'/' + utf8_str(fnm) + b'"'
TypeError: can't concat bytes to NoneType

3. test5.azw3

with python3:

Spoiler:

Unpacking raw markup language
Processing ncx / toc
Building an epub-like structure
Building proper xhtml for each file
Traceback (most recent call last):
File "kindleunpack.py", line 1016, in <module>
sys.exit(main())
File "kindleunpack.py", line 1004, in main
unpackBook(infile, outdir, apnxfile, epubver, use_hd)
File "kindleunpack.py", line 918, in unpackBook
process_all_mobi_headers(files, apnxfile, sect, mhlst, K8Boundary, False, ep
ubver, use_hd)
File "kindleunpack.py", line 835, in process_all_mobi_headers
processMobi8(mh, metadata, sect, files, imgnames, pagemapproc, k8resc, obfus
cate_data, apnxfile, epubver)
File "kindleunpack.py", line 531, in processMobi8
usedmap = htmlproc.buildXHTML()
File "\mobi_html.py", line 281, in bu
ildXHTML
replacement = b'"../' + pdir + b'/' + fnm + b'"'
TypeError: can't concat bytes to str

python 2:

Spoiler:

Unpacking raw markup language
Processing ncx / toc
Building an epub-like structure
Building proper xhtml for each file
Error: 'ascii' codec can't decode byte 0xe3 in position 79: ordinal not in range
(128)
Traceback (most recent call last):
File "kindleunpack.py", line 1004, in main
unpackBook(infile, outdir, apnxfile, epubver, use_hd)
File "kindleunpack.py", line 918, in unpackBook
process_all_mobi_headers(files, apnxfile, sect, mhlst, K8Boundary, False, ep
ubver, use_hd)
File "kindleunpack.py", line 835, in process_all_mobi_headers
processMobi8(mh, metadata, sect, files, imgnames, pagemapproc, k8resc, obfus
cate_data, apnxfile, epubver)
File "kindleunpack.py", line 531, in processMobi8
usedmap = htmlproc.buildXHTML()
File "mobi_html.py", line 286, in bu
ildXHTML
flowpart = b"".join(srcpieces)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 79: ordinal
not in range(128)

4. test2.azw3 with python 3(fixed):

Spoiler:

Unpacking raw markup language
Processing ncx / toc
Building an epub-like structure
Building proper xhtml for each file
Traceback (most recent call last):
File "kindleunpack.py", line 1016, in <module>
sys.exit(main())
File "kindleunpack.py", line 1004, in main
unpackBook(infile, outdir, apnxfile, epubver, use_hd)
File "kindleunpack.py", line 918, in unpackBook
process_all_mobi_headers(files, apnxfile, sect, mhlst, K8Boundary, False, ep
ubver, use_hd)
File "kindleunpack.py", line 835, in process_all_mobi_headers
processMobi8(mh, metadata, sect, files, imgnames, pagemapproc, k8resc, obfus
cate_data, apnxfile, epubver)
File "kindleunpack.py", line 557, in processMobi8
guidetext += cover.guide_toxml()
TypeError: Can't convert 'bytes' object to str implicitly

I have modied to intent to fix case 1 and 4.
Followings is the patch.

Spoiler:

diff -u nlib_3/kindleunpack.py nlib/kindleunpack.py
--- nlib_3/kindleunpack.py Wed Oct 8 18:37:26 2014
+++ nlib/kindleunpack.py Thu Oct 9 19:07:33 2014
@@ -473,7 +473,7 @@
k8proc.buildParts(rawML)

# collect information for the guide first
- guidetext = k8proc.getGuideText()
+ guidetext = unicode_str(k8proc.getGuideText())

# if the guide was empty, add in any guide info from metadata, such as StartOffset
if not guidetext and 'StartOffset' in metadata:
@@ -488,9 +488,9 @@
last_start = 0
seq, idtext = k8proc.getFragTblInfo(last_start)
filename, idtext = k8proc.getIDTagByPosFid(toBase32(seq), b'0000000000')
- linktgt = filename
- if idtext != '':
- linktgt += '#' + idtext
+ linktgt = unicode_str(filename)
+ if unicode_str(idtext) != '':
+ linktgt += '#' + unicode_str(idtext)
guidetext += '<reference type="text" href="Text/%s" />\n' % linktgt

# if apnxfile is passed in use it for page map information
diff -u nlib_3/mobi_cover.py nlib/mobi_cover.py
--- nlib_3/mobi_cover.py Wed Oct 8 18:37:26 2014
+++ nlib/mobi_cover.py Thu Oct 9 19:49:40 2014
@@ -233,4 +233,4 @@
text_dir = os.path.relpath(files.k8text, files.k8oebps)
data = '<reference type="cover" title="Cover" href="{:s}/{:s}" />\n'.format(\
text_dir, self.cover_page)
- return data.encode('utf-8')
+ return data #.encode('utf-8') Changed to unicode
diff -u nlib_3/mobi_k8proc.py nlib/mobi_k8proc.py
--- nlib_3/mobi_k8proc.py Wed Oct 8 18:37:26 2014
+++ nlib/mobi_k8proc.py Thu Oct 9 19:25:45 2014
@@ -246,7 +246,7 @@
m = re.search(svg_tag_pattern, flowpart)
if m is not None:
# svg
- type = b'svg'
+ type = 'svg'
start = m.start()
m2 = re.search(image_tag_pattern, flowpart)
if m2 is not None:

Take care,

[KevinH]

Hi tkeo,
Please note everything but file names and paths in mobi_k8proc.py must be bytes (bytestring), so as not to force the upconverting to unicode anyplace in that routine. So only filenames and path segments should have utf8_str on it and all literals should start with a b.

Since the type field was mixed sometimes 'inline' vs b'inline' or 'file vs b'file', this is what caused the None in the directory (it did not properly detect we had inlined the CDATA)

In mobi_cover.py, everything is converted to unicode asap then only the cover page xhtml should be encoded to utf-8 just before file creation.

I will check to make sure that this follows.

Thanks for testing. I will get a new version out after incorporating some of your changes but trying to keep to the rules I stated above.

Thanks,

KevinH

ps. I have attached a new version of nlib.zip below.

If this is no more stable than its predecessor, I am going to split out all binary routines into their own modules and convert to unicode at the boundary instead of trying to keep the code mostly the same as before.

It is just too hard to know what the data types are for each variable and so it gets confusing as to what should be converted to binary strings and what kept as unicode.

Right now most parts of mobi_header, mobi_index, mobi_dict, parts of mobi_ncx, parts of mobi_k8resc, mobi_k8proc, and most parts of mobi_html require some form of binary bytes code. Then kindleunpack has to deal with both types and convert properly when mixing them.

If I have to I will completely refactor the code to move all binary bytes processing to dedicated modules to try to prevent mixed data types in the same module completely.

I probably should have done that from the beginning but never dreamed it would be this hard to track down and fix all of the inconsistencies.

Thanks again for all of your hard work testing it and sending patches.

KevinH

[tkeo]

Hi Kevin,

The latest version of nls.zip unpack all my test ebooks without any errors!!

I will check unpacked data more carefully this weekend.

Thanks,

[KevinH]

Hi tkeo,

Great news! Hopefully, it will not mess anything up in the detailed tests.

Refactoring the code may not then be needed, but in the long run I think it is a good idea.

Thanks,

KevinH

Originally Posted by tkeo

Hi Kevin,

The latest version of nls.zip unpack all my test ebooks without any errors!!

I will check unpacked data more carefully this weekend.

Thanks,

[tkeo]

Hi Kevin,

I find bugs.

1.
All Style directories are empty and regarding css in content.opf are absent, when unpacking with python 3.

2.
File names in mobi7 directories are wrong.
book.html, content.opf and toc.ncx are in mobi7.


Take care,

[DiapDealer]

Originally Posted by tkeo

2.
File names in mobi7 directories are wrong.
book.html, content.opf and toc.ncx are in mobi7.

I believe that's as designed (albeit it's a change from previous versions). Why would it be seen as "wrong"?

With the old way of using the original file name for the various mobi7 files, it was possible for KindleUnpack to create invalid opf/ncx files (namely when ampersands were in the original file name).

Is there something that's been inadvertently overlooked/broken by using a similar naming convention as the KF8 routines?

[tkeo]

Hi,

Originally Posted by DiapDealer

I believe that's as designed (albeit it's a change from previous versions). Why would it be seen as "wrong"?

Oh, I have not noticed that.

I have compared with v0.75.
By v0.75 the filenames in mobi7 have the original filenames with extentions(html, opf and ncx).

Thanks,

[DiapDealer]

Originally Posted by tkeo

Oh, I have not noticed that.

I have compared with v0.75.
By v0.75 the filenames in mobi7 has the original filename with extentions(html, opf and ncx).

Thanks,

Ah, sorry... I had access to a pre-0.76 beta (before the python3-compatibility overhaul) where Kevin had made that change at my request/suggestion (and his agreement). I'd forgotten that no one had really seen that change yet. Do you foresee any issues with it?

[KevinH]

Originally Posted by tkeo

1.
All Style directories are empty and regarding css in content.opf are absent, when unpacking with python 3.

Hi,

Point 2 is expected behaviour as the file name was causing issues in opf with spaces and illegal characters. So I changed mobi7 to act similar to how mobi8 acts.

Point 1 is totally my fault. I changed the 'inline' and 'file' strings to be binary everplace but missed 1 place! In processMobi8 in kindleunpack.py, I should have changed things to test for b'file' instead of 'file'.

I will make than change, clean-up a number of non-parallelisms, and minor nits then we should be good to go.

Thanks again for all of your help and testing!!!!!

- edit -

I have put together what I hope is the first stable version: nlib_rev1.zip of what will eventually become the heart of KindleUnpack_v0.80.

If there are users of KindleUnpack that know how to use the command-line, please try unpacking your mobis and report any bugs with this version here.

I will move on to porting the gui frontend next. Much of the tk stuff has changed module names, so this should be fun!


Take care,

KevinH

[KevinH]

Hi tkeo,

I would like to merge your experimental support for unpacking audio and video into this new upcoming 0.80 version since that would make a very nice new feature to go along with the boring port changes.

I know you added patches for this. Do you want me to convert them to dual python 2 / python 3 use or do you want to do it?

Thanks,

KevinH

[KevinH]

Hi All,

Since I need a github tree to work on Sigil, I was thinking of adding in KindleUnpack as an additional project.

So does anyone know if there is quick way to initialize a github tree with copies of lots of release versions. Or do I literally have to start with the very first and commit it and then go version by version committing all 60 or so of them one at a time?

Thanks,

KevinH

[KevinH]

Hi All,

I would like to build a github repository with as full a history as possible for all of the official (non-testing, on-experimental) releases of KindleUnpack

I have scoured this Thread as well as all of my available sources and I have found the following official versions. If anyone has any other release versions they are willing to share I would love to have a complete set.

Thanks!

Kevin

mobiunpack_v011.py
mobiunpack_v012.py
mobiunpack_v014.py
mobiunpack_v015.py
mobiunpack_v016.py
mobiunpack_v017.py
mobiunpack_v018.py
mobiunpack_v019.py
mobiunpack_v020.py
mobiunpack_v021.py
mobiunpack_v022.py
mobiunpack_v023.py
mobiunpack_v024.py
mobiunpack_v026.py
mobiunpack_v027.py
mobiunpack_v028.py
mobiunpack_v029.py
mobiunpack_v030.py
mobiunpack_v031.py
mobiunpack_v032.py
Mobi_Unpack_v034.zip
Mobi_Unpack_v035.zip
Mobi_Unpack_v036.zip
Mobi_Unpack_v042.zip
Mobi_Unpack_v045.zip
Mobi_Unpack_v047.zip
Mobi_Unpack_v050.zip
Mobi_Unpack_v053.zip
Mobi_Unpack_v054.zip
Mobi_Unpack_v056.zip
Mobi_Unpack_v058.zip
Mobi_Unpack_v060.zip
KindleUnpack_v062.zip
KindleUnpack_v063.zip
KindleUnpack_v063a.zip
KindleUnpack_v064.zip
KindleUnpack_v065.zip
KindleUnpack_v066.zip
KindleUnpack_v067.zip
KindleUnpack_v070_experimental.zip
KindleUnpack_v071.zip
KindleUnpack_v072a.zip
KindleUnpack_v072b.zip
KindleUnpack_v073.zip
KindleUnpack_v075.zip
KindleUnpack_v076pre_release.zip
nlib_rev1.zip

[pdurrant]

I seem to have 0.12 (attached)

And possibly 0.14 and 0.15