KindleUnpack (MobiUnpack): Extracts text, images and metadata from Kindle/Mobi files

[KevinH]

Originally Posted by pdurrant

I seem to have 0.12 (attached)

And possibly 0.14 and 0.15

Hi Paul,

Great thanks. I also found versions .17 to .22 in a clone of our old sourceforge ebook_tools repository via Google.

Take care,

Kevin

[tkeo]

Hi Kevin,

Originally Posted by KevinH

I would like to merge your experimental support for unpacking audio and video into this new upcoming 0.80 version since that would make a very nice new feature to go along with the boring port changes.

I know you added patches for this. Do you want me to convert them to dual python 2 / python 3 use or do you want to do it?

I can modify the patch.
But I have not updated them since I posted; they are needed to make modify links in xhtmls. I have no idea to do so.
If you have the idea, could you convert them?

Thanks,

[KevinH]

Hi tkeo,
I'll try to integrate them.

Thanks,

Kevin

Originally Posted by tkeo

Hi Kevin,
I can modify the patch.
But I have not updated them since I posted; they are needed to make modify links in xhtmls. I have no idea to do so.
If you have the idea, could you convert them?

Thanks,

[KevinH]

Hi All,

In preparation for trying to build a github repository that has as complete a history as possible, I have collected all of the announcements and dates and added developer information as best as I could reconstruct from the discussions held in these threads.

I have included it here. Please look it over. If I have left anyone out, please let me know. When attribution was not clear, I simply used the term "Developer Team". I am happy to make any additions or corrections. Just let me know.

I will then use this history to build the git repositiory for KindleUnpack and see it with the list of found releases I published earlier.

Thanks,

Kevin

v011
2009-11-12
Adam Selene
Initial Version

I reimplemented huff/cdic compression in Python, and did a few other things while I was at it. The new script:

* decompresses about 25x faster than mobihuff.py
* uses much less memory (about 16x on my largest test file)
* implements conversion of uncompressed and Palmdoc-compressed files
* handles trailing data correctly in all cases

Check it out: http://www.mit.edu/afs/athena/user/m.../mobiunpack.py
===

v012
2010-02-05
pdurrant
added ability to output images, and all into one folder
===

v013
2010-02-06
pdurrant
added back in optional output dir for those who don't want it based on infile
===

v014
2010-02-07
KevinH
auto flush stdout and wrapped in main, added proper return codes
===

v015
2010-02-08
KevinH
added support for metadata
===

v016
2010-02-11
pdurrant
metadata now starting to be output as an opf file
===

v017
2010-02-17
pdurrant
Additional metadata read and output, rawml tweaked to input html for Mobipocket Creator
===

v018
2010-02-18
KevinH
removed raw mobi file completely but kept _meta.html file for ease of conversion
===

v019
2010-02-18
pdurrant
added in metadata for ASIN, Updated Title and Rights to the opf

Added in standard (OPF 1.01) metadata Rights. Added in non-standard items ASIN 
and Updated Title. If we can find out where they should really go in an OPF to 
get put into a compiled Mobipocket file, we should move them. Mobipocket Creator 
ignores them where they are
===

v020
2010-02-18
KevinH
remove _meta.html since no longer needed
===

v021
2010-03-13
pdurrant
Fixed typos in the opf, updated handling of test for trailing data/multibyte characters
===

v022
2010-04-12
pdurrant
Fixed problem with > 9 images
===

v023
2010-08-29
pdurrant
Now output Start guide item support to opf
===

v024
2011-05-18
pdurrant
Set firstaddl value for 'TEXtREAd'
===

v025
2011-05-18
pdurrant
Now added character set metadata to html file for utf-8 files.
===

v026
2011-06-27
siebert
Dictionary support added. Image handling speed improved.

For huge files create temp files to speed up decoding.
Language decoding fixed. Metadata is now converted to utf-8 when written to opf file.
===

v027
2011-07-18
siebert
Add idx:entry attribute "scriptable" if dictionary contains entry length tags.

Don't save non-image sections as images. Extract and save source zip file
included by kindlegen as kindlegensrc.zip.
===

v028
2011-07-19
KevinH
Added back correct image filename extensions, created FastConcat class to simplify and clean up
===

v029
2011-07-23
siebert
Metadata handling reworked, multiple entries of the same type are now supported.

Several missing types added.
FastConcat class has been removed as in-memory handling with lists is faster, even for huge files.
===

v030
2011-08-05
KevinH
Add support for outputting **all** metadata values

- Cover ThumbNail Image
- Text to Speech Disabled Flag
- Font Signature encoded as hex
- Tamper Proof Keys encoded as hex
- All unknown metadata keys as hex strings
===

v031
2011-09-01
pdurrant
Now supports Print Replica ebooks, outputting PDF and mysterious data sections
===

v032
2011-10-27
fandrieu,siebert,KevinH,DiapDealer
Now supports NCX file extraction/building.

Overhauled the structure of mobiunpack to be more class oriented.
===

v033
2011-12-23
KevinH
Split Classes into separate files and added prelim support for KF8 format eBooks

main program renamed to mmobi_unpack.py to indicate KF8 support
===

v034
2012-01-18
KevinH, nickredding, DiapDealer
Improved KF8 support, guide support, bugs fixed

- added support for image files used in CSS sheets (needed for fixed layout)
- modified the unpacker to deal with the extra metadata fields used by 
    fixed-layout ebooks "RegionMagnification", "fixed-layout", "book-type", 
    "orientation-lock", "original-resolution" 
- identified the BOUNDARY section number
- builds the epub from the K8 pieces with compression now 
- fixed the mobi_k8proc.py class code to be better encapsulated (added accessor methods)
- fixed support for older mobis with no ncx
===

v035
2012-01-27
nickredding
Added splitting combo mobi7/mobi8 into standalone mobi7 and mobi8 files

Also handle mobi8-only file properly
===

v036
2012-01-27
nickredding, KevinH
very minor changes to support KF8 mobis with no flow items, no ncx, etc
===

v037
2012-01-29
KevinH
separate output, add command line switches to control, interface to Mobi_Unpack.pyw
===

v038
2012-01-29
Developer Team
improve split function by resetting flags properly, fix bug in Thumbnail Images
===

v039
2012-01-29
Developer Team
improve split function so that ToC info is not lost for standalone mobi8s
===

v040
2012-01-29
Developer Team
improve mobi7 split, add graphic novel metadata, improve debug for KF8
===

v041
2012-01-29
Developer Team
fix when StartOffset set to 0xffffffff, fix to work with older mobi versions, fix metadata bugs
===

v042
2012-02-12
Developer Team
add new class interface to allow it to integrate more easily with internal calibre routines
===

v043
2012-02-20
Developer Team
bug fixes for new class interface
===

v044
2012-02-29
Developer Team
more bug fixes and fix for potnetial bug caused by not properly closing created zip archive
===

v045
2012-03-08
KevinH, DiapDealer
sync to version in the new Mobi_Unpack plugin
===

v046
2012-03-08
Developer Team
minor opf improvements
===

v047
2012-03-23
pdurrant, DiapDealer, KevinH
fixes for: obfuscated fonts, improper toc links and ncx, add support for opentype fonts
===

v048
2012-04-22
Developer Team
ncx link fixes
===

v049
2012-04-23
KevinH
use azw3 when splitting mobis, mobi_split bug fixes
===

v050
2012-04-26
kovidgoyal
fixed bug with unreliable decoding of zlib header, fixed image bug in urls in inline styles
===

v051
2012-06-04
pdurrant, DiapDealer
fix for converting filepos links to hrefs, Added GPL3 notice, made KF8 extension just '.azw3'
===

v052
2012-07-21
pdurrant, DiapDealer
fix for cover metadata (no support for Mobipocket Creator)
===

v053
2012-07-25
KevinH
fix header flag for proper identification of embedded fonts, added new metadata items
===

v054
2012-09-13
DiapDealer
Added error-handling so wonky embedded fonts don't bomb the whole unpack process

also entity escape KF8 metadata to ensure valid OPF.
===

v055
2012-10-21
pdurrant
Strip extra StartOffset EXTH from the mobi8 header when splitting

keeping only the relevant one. For mobi8 files, don't generate duplicate guide 
entries from the metadata if we could extract one from the OTH table.
===

v056
2012-10-26
DiapDealer
Added further entity escaping of OPF text.

Allow unicode string file paths to be passed as arguments to the unpackBook 
method without blowing up later when the attempt to "re"-unicode a portion of 
that filename occurs in the process_all_mobi_headers method.
===

v057
2012-10-27
pdurrant
Fixed eror when splitting Preview files downloaded from KDP website
===

v058
2012-10-27
DiapDealer
Output original kindlegen build log ('CMET' record) if included in the package.
===

v059
2012-11-08
pdurrant
Include and extend functionality of DumpMobiHeader, replacing DEBUG with DUMP

Much added DUMP functionality, including full dumping and descriptions of sections
===

v060
2012-12-10
nleblanc88, Sergey Dubinets, KevinH
Bug fixes in opf, div tables, bad links, page breaks, section descriptions

- fixs for file/paths that require full unicode to work properly
- replace subprocess with multiprocessing to remove need for unbuffered stdout
===

v061
2013-01-17
KevinH, DiapDealer
renamed to be KindleUnpack and better unicode/utf-8 path bug fixes and other minor fixes
===

v062
2013-04-22
KevinH, DiapDealer
bug fix release

- fix for improved command line handling on Windows boxes under multiprocessing
- improvements to split to allow it to work with newer Kindle Previewers
- improvements to split to allow it to replace fonts with zero-szed sections in Mobi 7
- improvements to opf to prevent html tags from interfering with opf xml creation
===

v063
2014-02-09
tkeo
Modified to process right to left page progression books properly.

- Added some id_map_strings and RESC section processing; metadata and
    spine in the RESC are integrated partly to content.opf.
- imghdr fixes
===

v063a
2014-02-16
tkeo
Separated K8 RESC processor to an individual file. Bug fixes. Added cover page creation.
===

v064
2014-02-23
KevinH, NiLuJe
minor bug fixes to more properly handle unicode command lines, and support for more jpeg types
===

v064a
2014-04-10
tkeo, DiapDealer
Modifed to safely handle some irregular mobi and azw3 files, parsing RESC fixes
===

v064b
2014-04-11
tkeo
Modifed to create k8resc.spine for no RECS files. allow cover page creation when no RESC
===

v065
2014-04-26
DiapDealer
Bug fixes to shorten title and remove epub3 "properties" to make the output epub2 compliant
===

v065a
2014-05-04
tkeo
Bug fixes to extract RESC section correctly, to prevent item id confliction

add support to process multiline comments in RESC.
===

v066
2014-05-18
tkeo, KevinH
Bug fix to deal with missing first resource information sometimes generated by calibre
===

v066a
2014-05-23
tkeo
Fixed minor bugs, which probably do not affect the output anything
===

v067
2014-06-10
KevinH
Fixed Mobi Split functionality bug with azw3 images not being properly copied
===

v068
2014-06-12
KevinH
added preliminary support for handling PAGE sections to create page-map.xml
===

v069
2014-06-12
KevinH
added preliminary support for CONT and CRES for HD Images
===

v070
2014-06-14
KevinH
added preliminary support for decoding apnx files when used with azw3 ebooks
===

v071
2014-06-17
KevinH, DiapDealer, tkeo
extensive refactoring of kindleunpack.py to make it more manageable
===

v072
2014-06-18
tkeo, DiapDealer, KevinH
many bug fixes from tkeo: fix pageProcessing, fix print replica, fix resc usage, fix font mangling, etc.
===

v072a
2014-06-18
DiapDealer
fix for still broken PrintReplica support
===

v072b
2014-06-22
tkeo
preview for primary epub3 support. A parameter epubver(default='2') is added
===

v072c
2014-06-24
KevinH
added preview for apnx page support
===

v072d
2014-06-25
DiapDealer, tkeo
more bugs fixed in preview features

much improved GUI with ability to dynaically grow the Log Window with preference support
===

v072e
2014-06-25
DiapDealer, KevinH
more bug fixes, Tk GUI adds support for epub version and HDImage use
===

v072f
2014-06-26
DiapDealer, KevinH
more bug fixes, much improved GUI, implement use hd images if present
===

v072g
2014-06-28
tkeo
minor bug fixes in cover image renaming and cleanups
===

v072h
2014-07-05
tkeo
proposed changes for RESC processing and code simplification
===

v072x
2014-07-08
KevinH
experimtental RESC conversion to mobi_resc.py based on old mobi2xhtml.py

updated mobi_header and mobi_k8proc to use the correct fragment and guide terms
in place of div and other to better match the terms that both Calibre and Amazon 
use internally to their own software.
===

v072y
2014-07-10
KevinH, tkeo
more k8resc changes to simplify integration of epub3 support
===

v072z
2014-07-11
tkeo, KevinH
remove redundancy in mobi_opf.py and bug fixes for mobi_k8resc.py
===

v073
2014-07-15
tkeo, KevinH
faster mobi split, numerous bug fixes in mobi_k8proc, mobi_header, mobi_opf, mobi_k8resc, etc
===

v074
2014-07-30
tkeo
added refines metadata, fixed language code in ncx and title in nav, added support for opf: from refines
===

v075
2014-09-13
KevinH
much improved dictionary support including support for multiple inflection sections, minor mobi_opf fixes

[pdurrant]

Wow. Impressive work! It will be interesting to browse the archive once it's done.

There was a short-lived 0.11pd, but since all those changes appeared in 0.12, perhaps it's best forgotten.

[KevinH]

Hi,

One quick question if there are any git experts our there. I understand how to set the historic date and include the historic commit messages and even provide the authors in the commit message itself, but for some reason git will not allow me to actually set the commit author to anyone who does not have his/her own git account username and password.

Is there some way to override the setting of author that gets into the commit history?

If not, I will just clearly add the authors information to the commit message so that the proper author gets credit.

Thanks,

KevinH

[AcidWeb]

Well bare GIT allow to set commit author with --author "New Author Name <email@address.com> but I doubt that GitHub will accept commit from user that is not connected with any GitHub account.

Also: GIT. At last.

[KevinH]

Hi All,

Okay I have tried to create as much of the history as I could remember, find on the web, or find in this thread.

It is not complete. Some of the releases are missing from the history but I have folded those updates into the next release I was able to find, and credited all changes.

So please give this a look see:

I have not committed the dual python 2 / python 3 support, so this repository ends with an as of today version of KindleUnpack_v076.

If all are okay with repository, I will push all of the dual python 2 and 3 support into the tree once I have working versions of the front-end.

The url for the repository is:

https://github.com/kevinhendricks/KindleUnpack

Feedback welcome. I now have to figure out how to give other current contributors: (ie. pdurrant, DiapDealer, and tkeo) direct access to the repository. If anyone knows how to do this please let me know.

Thanks,

KevinH

[eschwartz]

Originally Posted by AcidWeb

Well bare GIT allow to set commit author with --author "New Author Name <email@address.com> but I doubt that GitHub will accept commit from user that is not connected with any GitHub account.

Also: GIT. At last.

This should work just fine.

And GitHub does not restrict all users from accepting commits made by non-GitHub users.
It simply will not have a clickable link to the GitHub user on the commit reference.
And GitHub identifies users based on their email address.
See this recent calibre commit for an example of an unidentified author: https://github.com/kovidgoyal/calibr...97696d39a7a0f2
chaley did this, (Charles Haley <cbhaley@i.wont.say.com>) but his fake email address isn't listed as his on his account, so it's an unknown user.

Originally Posted by KevinH

Hi,

One quick question if there are any git experts our there. I understand how to set the historic date and include the historic commit messages and even provide the authors in the commit message itself, but for some reason git will not allow me to actually set the commit author to anyone who does not have his/her own git account username and password.

Is there some way to override the setting of author that gets into the commit history?

If not, I will just clearly add the authors information to the commit message so that the proper author gets credit.

Thanks,

KevinH

git or GitHub? how are you doing it?

You need to specify both a name and email. But the email can be left blank, by specifying

--author "FN LN <(someone@email.address has been OMITTED)>"
aka
--author "FN LN <>"

Originally Posted by KevinH

Feedback welcome. I now have to figure out how to give other current contributors: (ie. pdurrant, DiapDealer, and tkeo) direct access to the repository. If anyone knows how to do this please let me know.

https://help.github.com/articles/add...al-repository/

[AcidWeb]

@eschwartz - I stand corrected.

@KevinH - You can store Python2/3 version in separate branch until code is stable. It is another GIT aspect that you should explore.

Also - First line of commit message. It is customary to capitalize it and not exceed 50 characters.

[KevinH]

Hi All,

Thanks for your helpful comments. I wish I had known them before yesterday when I spent 4 hours hand committing and dating each of the commits!!

So is there an easy way to hand edit the history itself without having to rebuild, and recommit almost 60 versions again?

Thanks,

KevinH

Yes, I will branch the python 2/3 unified version until it becomes stable and is released.

[eureka]

Originally Posted by KevinH

So is there an easy way to hand edit the history itself without having to rebuild, and recommit almost 60 versions again?

Maybe you are looking for git-filter-branch?

Originally Posted by git help filter-branch

Lets you rewrite Git revision history by rewriting the branches mentioned in the <rev-list options>, applying custom filters on each revision. Those filters can modify each tree (e.g. removing a file or running a perl rewrite on all files) or information about each commit. Otherwise, all information (including original commit times or merge information) will be preserved.

Yes, it will let you change commit author without changing committer. Here is a relevant (but a bit different) example provided by GitHub.

While it's obvious, I'd like to add that when history will be rewritten, you will have a different "repository" (in comparison with repository on GitHub and all previous clones of it). As it's mostly an organizational and not technical problem, you may want to delete and re-create repository on GitHub (though it's optional). EDIT: that GitHub example shows also how to forcefully push changed history without re-creating repository.

In addition to AcidWeb words, reference format (style) of Git commit messages is described here. Message format isn't dictated by Git itself, but linked post is a good format example or starting point for making your own style.

UPD: JFYI (and for sake of historical correctness), URL in the very first commit message is wrong, it literally have three dots in it:

http://www.mit.edu/afs/athena/user/m.../mobiunpack.py

[KevinH]

HGi,

Thanks for the links. Unfortunately there is no way to use an e-mail replacement algorithm since there were no e-mails for any of the authors.

So the credit for authorship is in the long commit message itself.

So I may be better off simply redoing the damn 60 commits by hand and completely rebuilding it from scratch.

Once I have time this weekend I will try that instead.

As for capitalizing and keeping to 50 chars, I simply don't think any of that is actually important. I hate to count chars.

Thanks everyone,

It seems to be I need to go back to the drawing board.

KevinH

Originally Posted by eureka

Maybe you are looking for git-filter-branch?

Yes, it will let you change commit author without changing committer. Here is a relevant (but a bit different) example provided by GitHub.

While it's obvious, I'd like to add that when history will be rewritten, you will have a different "repository" (in comparison with repository on GitHub and all previous clones of it). As it's mostly an organizational and not technical problem, you may want to delete and re-create repository on GitHub (though it's optional). EDIT: that GitHub example shows also how to forcefully push changed history without re-creating repository.

In addition to AcidWeb words, reference format (style) of Git commit messages is described here. Message format isn't dictated by Git itself, but linked post is a good format example or starting point for making your own style.

UPD: JFYI (and for sake of historical correctness), URL in the very first commit message is wrong, it literally have three dots in it:

http://www.mit.edu/afs/athena/user/m.../mobiunpack.py

[eureka]

Originally Posted by KevinH

Unfortunately there is no way to use an e-mail replacement algorithm since there were no e-mails for any of the authors.

These excerpts worked for me:

export GIT_AUTHOR_NAME="Adam Selene"
export GIT_AUTHOR_EMAIL=""

or

export GIT_AUTHOR_NAME="Adam Selene"
export GIT_AUTHOR_EMAIL="unknown.so.do.not.reply@host.invalid"

.invalid is a reserved invalid TLD according to RFC2606.

And you may change commit message also with git-filter-branch. Or use git rebase -i.

[eschwartz]

Yes. Rebasing will allow you to walk through each commit and change the metadata.

# Using "--root" is very important, you cannot simply specify the sha1 of the
# first commit, or it will start with the second commit
git rebase -i --root
:%s/pick/edit/g
:wq
git commit --amend --author="Adam Selene <>" && git rebase --continue
...
...

EDIT: Never mind, in my test it seems as though amending a commit makes it more fussy.
fatal: Malformed ident string: 'Adam Selene <> 1268438400 +0000'

EDIT #2: use this to "amend" a commit by undoing the commit, recommitting using the saved EDITMSG, then continuing rebase:

git undo-commit && git commit --author="Adam Selene <>" -m "$(cat .git/COMMIT_EDITMSG)" && git rebase --continue
...
...

Assumes you have created the following alias (should be git builtin):

# revert "git commit", leaving you with staged changes.
git config --global alias.undo-commit 'reset --soft HEAD^'