KindleUnpack (MobiUnpack): Extracts text, images and metadata from Kindle/Mobi files

[tkeo]

Hi,

I have fixed bugs regarding RESC section processing. Three bugs are fixed as follows:

1. In 'kindleunpack.py', RESC section extraction failure (reported before) is fixed.
2. In 'mobi_opf.py', potential confliction of item ids between the id generated in and extracted one from RESC is fixed.
3. In 'mobi_resc.py', modified to make possible to process multi-line comments in RESC.

Attached files are as follows,

• kindleUnpack_diff_v65a.zip: updated files (kindleunpack.py, mobi_opf.py and mobi_resc.py)
• kindleunpack_v65a.patch.txt: patch file version of the update (changes are identical to kindleUnpack_diff_v65a.zip)
• Illformed_RESC_exmaple.mobi and example4_src.zip: the example to comfirm fixes 2 and 3. It is made to conflict id intentionally and including multi-line comments in an opf file.

Thanks,
tkeo

[tkeo]

Hi,

Originally Posted by KevinH

At some point soon we should probably think about how to address epub3 vs epub2 features in some sane way since kindlegen seems to be moving to adopting epub3 compatibility faster than actual epub devices.

I have been thinking this matter.

I briefly summarize differences from epub2 to eupb3 related closely to KindleUnack. Please correct if it is wrong.

• In opf:
  • package version.
  • metadata tag and allowed attributes in its component.
  • allowed attributes in item tag in manifest.
  • allowed attributes in spine tag and item tag in the spine.
  • tours tag is removed.
• toc.ncx to toc in nav.xhtml.
• guide tag in opf is move to landmark in nav.xhtml.
  
• javascript and other contents support.

So, most chages to need is 'mobi_opf.py' and 'kindleunpack.py', and addition of nav.xhtml creation parser.
I think addition of parameter to indicate taget epub version to functions are necessary. And throgh consideration, questions are come into my mind.

How many target versions are necessary?

strict epub2, epub2+alpha(current version), epub3?

or

epub2+alpha(current version) and epub3 is enough?

Regarding to modify 'mobi_opf.py', which is better,

switching internal process in WriteOPF()

or

separeting WriteOPF() for each version?
The switching internal process is possible but it will decrease readability and may cause bugs to all versions...

I would like to hear opinions.
Thanks,

[KevinH]

Hi,

I am tied up and traveling for the next 2 weeks, so if someone wants to make a new official KindleUnpack release, please go ahead and do so. If not, upon my return I will do so.

Thanks,

KevinH

Originally Posted by tkeo

Hi,

I have fixed bugs regarding RESC section processing. Three bugs are fixed as follows:

1. In 'kindleunpack.py', RESC section extraction failure (reported before) is fixed.
2. In 'mobi_opf.py', potential confliction of item ids between the id generated in and extracted one from RESC is fixed.
3. In 'mobi_resc.py', modified to make possible to process multi-line comments in RESC.

Attached files are as follows,

• kindleUnpack_diff_v65a.zip: updated files (kindleunpack.py, mobi_opf.py and mobi_resc.py)
• kindleunpack_v65a.patch.txt: patch file version of the update (changes are identical to kindleUnpack_diff_v65a.zip)
• Illformed_RESC_exmaple.mobi and example4_src.zip: the example to comfirm fixes 2 and 3. It is made to conflict id intentionally and including multi-line comments in an opf file.

Thanks,
tkeo

[KevinH]

Hi,

My vote goes to separate writeOPF routines with as many common functions and methods pulled out to be shared to make it easier to maintain the pair.

A command line switch to enable epub3 features sounds good as well.

I would stick with what we have now for epub2, and going with official epub3.

If the need arises we can add a strict flag to the command line to rule out any hybrid approach.

My 2 cents ...

KevinH

Originally Posted by tkeo

Hi,



I have been thinking this matter.

I briefly summarize differences from epub2 to eupb3 related closely to KindleUnack. Please correct if it is wrong.

• In opf:
  • package version.
  • metadata tag and allowed attributes in its component.
  • allowed attributes in item tag in manifest.
  • allowed attributes in spine tag and item tag in the spine.
  • tours tag is removed.
• toc.ncx to toc in nav.xhtml.
• guide tag in opf is move to landmark in nav.xhtml.
  
• javascript and other contents support.

So, most chages to need is 'mobi_opf.py' and 'kindleunpack.py', and addition of nav.xhtml creation parser.
I think addition of parameter to indicate taget epub version to functions are necessary. And throgh consideration, questions are come into my mind.

How many target versions are necessary?

strict epub2, epub2+alpha(current version), epub3?

or

epub2+alpha(current version) and epub3 is enough?

Regarding to modify 'mobi_opf.py', which is better,

switching internal process in WriteOPF()

or

separeting WriteOPF() for each version?
The switching internal process is possible but it will decrease readability and may cause bugs to all versions...

I would like to hear opinions.
Thanks,

[eureka]

Calibre writes 0xffffffff in "First image index" field of MOBI header (it's a field at offset 108) when ebook-convert --mobi-file-type new is used and no images are provided in input file. Source code of this Calibre action is located at calibre/ebooks/mobi/writer8/mobi.py.

Kindle Touch 5.3.7 displays error ("Unable to Open Item") on selecting book for reading, when result of aforementioned ebook-convert is copied to device with .mobi extension, but when resulting file is copied to device with .azw3 extension, KT displays it without errors.

So, device is able to read resulting file, however, KindleUnpack v65 can't unpack it, showing following exception:

Spoiler:

Traceback (most recent call last):
  File "/tmp/KindleUnpack_v65/lib/kindleunpack.py", line 1699, in <module>
    sys.exit(main())
  File "/tmp/KindleUnpack_v65/lib/kindleunpack.py", line 1688, in main
    unpackBook(infile, outdir)
  File "/tmp/KindleUnpack_v65/lib/kindleunpack.py", line 1619, in unpackBook
    process_all_mobi_headers(files, sect, mhlst, K8Boundary, False)
  File "/tmp/KindleUnpack_v65/lib/kindleunpack.py", line 1143, in process_all_mobi_headers
    for i in xrange(beg, end):
OverflowError: Python int too large to convert to C long

Here are inputs of ebook-convert:

Spoiler:

• converting command:

  ebook-convert index.html without_images.mobi --title Without images --authors Anonymous --language en --mobi-file-type new

• input file (index.html):

  <!DOCTYPE html>
  <html>
    <head><title>Hello world!</title></head>
    <body><p>Hello world!</p></body>
  </html>

• Calibre version:

  $ ebook-convert --version
  ebook-convert (calibre 1.25.0)
  Created by: Kovid Goyal <kovid@kovidgoyal.net>

ebook-convert output is attached to this post (it's the same file but with two different extensions, just for convenience).

[KevinH]

Hi eureka,
Thanks for the bug report and test case. I now have a fix for this bug in my tree. I am away from my home traveling, and will integrate tkeo's fixes and your bug fix when I return in about a week.

Hope that works for you.

Take care,

KevinH

[KevinH]

Hi All,

As promised, here is KindleUnpack_v66 which is a bug fix release. It has all of the bug fixes from tkeo's v65a files:

1. RESC section extraction failure (reported before) is fixed. (tkeo)

2. potential confliction of item ids between the id generated in and extracted one from RESC is fixed. (tkeo)

3. now able to process multi-line comments in RESC. (tkeo)

plus an additional bug fix as reported by eureka:

4. fix for calibre generated mobis and azw3's with no images that have incorrect firstresource offsets set to missing 0xffffffff that azw3's ignore (KevinH)

Please give it a try and let us know of any further issues.

Thanks,

KevinH

[quiris]

And here you are detailed changeset between 0.65 and 0.66: https://github.com/quiris11/KindleUn...fe44c298ab8afe

[KevinH]

Hi,

If people would prefer unified diffs be posted as well for each release, that is simple enough to do. Just let me know.

KevinH

[tkeo]

Hi,

I have fixed minor bugs. They are found in testing the files attached in

Originally Posted by eureka

Calibre writes 0xffffffff in "First image index" field of MOBI header (it's a field at offset 108) when ebook-convert --mobi-file-type new is used and no images are provided in input file. Source code of this Calibre action is located at calibre/ebooks/mobi/writer8/mobi.py.

There were bugs; however, probably those bugs did not affect the output anything I think.

[tkeo]

Hi All,

I have modified KindleUnpack to support epub3.

This is not completed it yet. I am not satisfied with the method to switch the output epub version; so, I am considering to change it to bit flags or something in order to increase flexiblity for future extentions.

Bug reports and suggestions are welcomed.

Major modifications are as follows,

• Added mobi_cover.py, mobi_nav.py and mobi_taglist.py.
• A parameter epubver(default=2) is added to process_all_mobi_headers(), unpackBook() and writeOPF().
• In mobi_opf.py, build functions of package parts are extracted from writeOPF().

Detailed changes are in the following:

Spoiler:

KindleUnpack v0.66ex preview for primary epub3 support.
Change log.

Added mobi_cover.py, mobi_nav.py and mobi_taglist.py.

In kindleunpack.py,
A parameter epubver(default=2) is added to process_all_mobi_headers() and unpackBook().
'--epub3' is added to the command line options.
Modified process_all_mobi_headers() to support epub3: creating an epub3 opf and a navigation document.
Modified not to rename the cover image filename but to write out as coverXXXXX.ext.
Modified to find a cover page in k8proc instead of files.

In mobi_cover.py,
The CoverProcessor class is moved from mobi_resc.py.
DiapDealer's svg wrapper version is merged.

In mobi_nav.py,
The NAVProcessor class is created to support the navigation document of epub3.

In mobi_opf.py,
Build functions of package parts are extracted from writeOPF().
A parameter epubver(default=2) is added to writeOPF().
A parameter navname(default=None) is added to __init__().
Added functions for epub3.
Modified to remove same tags which are generated by kindleunpack previously.

In mobi_resc.py,
Modified the K8RESCProcessor class for clarity:
The Spine class is added and moved functions from the K8RESCProcessor;
a spine object is added as a member of the K8RESCProcessor.
Functions create() and getItemidList() in which routines are pulled out from
kindleunpack.py and mobi_opf.py respectively, are added to the Spine.
The function metadata_toxml() is modified to output comments
in order to remove same tags which are generated by kindleunpack previously.
The CoverProcessor class is moved to mobi_cover.py.
The minidom version of the K8RESCProcessor class is removed.

[R. Scot Johns]

Hello all, just a quick question here.

Aaron Shepherd is reporting that he is seeing an azw3 and converted mobi file inside the extracted contents of a mobi file, along with the kindlegensrc.zip file and the standard mobi7/mobi8 folders, but he is no longer finding images in the mobi8 folder, only in the mobi7 one.

However, I am not seeing any of this myself, just the two folders, the source zip archive and the log file. Although the file sizes show a disparity which indicates the possibility of a missing file of roughly the same size as the kindlegensrc file.

Is there a logical explanation for this? He is on a Mac and I'm on Windows 7 if that matters. He is also using the AppleScript and obviously I am not, although I doubt that has anything to do with it.

Is anyone else getting the azw3/mobi files in an extracted archive?

[KevinH]

Hi,

Sorry I am a bit confused by your question.

When kindlegen is run without the "do not include source" command line option, it automagically includes the zip of the source. It always includes both a mobi7 (old format) and mobi8 (aka kf8/azw3) version of the book in the palm container (.mobi file).

When KindleUnpack is run on this file it will try to output the source zip, will unpack the older mobi7 part of the file, and unpack the mobi8 (kf8) part of the file.

It will unpack the mobi7 part into the single large old html file and folder of images.

It will try and unpack the mobi8 (kf8) part into a typical epub-like structure with multiple html files, css, images, and etc. The resulting epub may not be valid but most times will be if kindlegen is passed in a valid epub.

KindleUnpack also has a command line option to allow splitting of the original file into separate old version (mobi7) and newer mobi 8 (azw3) files.

So what exactly are you seeing? What parts are missing? Did you add the "don't include source" option when running Kindlegen? What command line options did you provide to KindleUnpack?

If you post a test case that illustrates the issue, I would be happy to look at it.

KevinH

Originally Posted by R. Scot Johns

Hello all, just a quick question here.

Aaron Shepherd is reporting that he is seeing an azw3 and converted mobi file inside the extracted contents of a mobi file, along with the kindlegensrc.zip file and the standard mobi7/mobi8 folders, but he is no longer finding images in the mobi8 folder, only in the mobi7 one.

However, I am not seeing any of this myself, just the two folders, the source zip archive and the log file. Although the file sizes show a disparity which indicates the possibility of a missing file of roughly the same size as the kindlegensrc file.

Is there a logical explanation for this? He is on a Mac and I'm on Windows 7 if that matters. He is also using the AppleScript and obviously I am not, although I doubt that has anything to do with it.

Is anyone else getting the azw3/mobi files in an extracted archive?

[R. Scot Johns]

Okay, I have never used the split command, so that is where the .azw3/.mobi files are being produced upon extraction. I see that now. Cool.

Our disparity lies in the fact that Aaron says he is no longer getting images in the mobi8 folder when extracted, while I still am. We both still find images in the mobi7 folder.

The source file is, of course, only present when the "-dont_append_source" tag is not added to the command line. I am aware of that.

The reason for the query, aside from the missing images in the mobi8 folder, is because we're trying to ascertain how Amazon is sourcing the higher resolution images for the HD devices, now that a 5 Mb per image file size is allowed. Do these come from the kindlegensrc.zip archive, or the .azw3 file?

Mainly I was trying to account for what is in the final converted file, and the missing images are an unknown variable. So my question is whether anyone else is seeing an empty images folder in the mobi8 section of the unpack? Or is this an anomaly?

For the record, here is what Aaron is seeing (and this is in an unpacked version of a preview file downloaded from KDP):

Zipped submission archive (with uncompressed images) [kindlegensrc.zip]
Mobi7 source folder (with compressed images)
Mobi7 ebook (with compressed images)
Mobi8 source folder (no images!) ***
AZW3 ebook (uncompressed images)
Kindlegen log

Thanks for the reply!

[KevinH]

Hi,

Our disparity lies in the fact that Aaron says he is no longer getting images in the mobi8 folder when extracted, while I still am. We both still find images in the mobi7 folder.

Then he should post a sample ebook that illustrates this problem and send me a link to it (use PM if you want privacy) so that I can recreate the issue and get it fixed.

The reason for the query, aside from the missing images in the mobi8 folder, is because we're trying to ascertain how Amazon is sourcing the higher resolution images for the HD devices, now that a 5 Mb per image file size is allowed. Do these come from the kindlegensrc.zip archive, or the .azw3 file?

Neither. There is now a third part of the kindlegen generated file added after yet again another "separator" that is called an HD CONTAINER that stores all of the HD images, one per sector of the palm file.

I am not sure that anyone has bothered to decode or dump those as they are only produced by the latest versions of kindlegen. There is a program called DumpMobiHeader_v14.py someplace I wrote that can show you where they are being stored in the kindlegen generated mobi file. I will loook and see if I have a later version of it and post it for you to help you find these HD container files.

Mainly I was trying to account for what is in the final converted file, and the missing images are an unknown variable. So my question is whether anyone else is seeing an empty images folder in the mobi8 section of the unpack? Or is this an anomaly?

There is no such thing as an anomaly, just a bug. If you can please post a link to a sample ebook that illustrates this issue, I will try to get it fixed.

For the record, here is what Aaron is seeing (and this is in an unpacked version of a preview file downloaded from KDP):

Zipped submission archive (with uncompressed images) [kindlegensrc.zip]
Mobi7 source folder (with compressed images)
Mobi7 ebook (with compressed images)
Mobi8 source folder (no images!) ***
AZW3 ebook (uncompressed images)
Kindlegen log

What do you mean "compressed" images versus uncompressed images? This could be the source of the problem but I need a sample ebook that illustrates the issue if you want it fixed.

Take care,

KevinH