KindleUnpack (MobiUnpack): Extracts text, images and metadata from Kindle/Mobi files

[naisren]

I sent you a message, and thanks for your attention.
my steps,
1, unpack the dic mobi.
2, Using the new opf to repack it without editing anything, just for proving everything is back in its own way. During it, warnings about invalid images came out, I checked the so called invalid images, and tried to open them with picture tool and failed. Many images are OK without warning.
3, After the new mobi is generated, I opened it using PC mobi reader, and many functions refused to work.

[siebert]

Originally Posted by naisren

1, unpack the dic mobi.
2, Using the new opf to repack it without editing anything, just for proving everything is back in its own way. During it, warnings about invalid images came out, I checked the so called invalid images, and tried to open them with picture tool and failed. Many images are OK without warning.
3, After the new mobi is generated, I opened it using PC mobi reader, and many functions refused to work.

I've tried the same with the file you've send, but my experience is quite different. The unpacking with mobiunpack and repacking with mobigen works fine, no complains about images from both programs besides the common warning from mobigen that the cover image is too small.

The repacked mobi works fine in the kindle app (including images), but without the javascript functions as javascript is unsupported by the kindle app.

The mobipocket reader crashes when I try to open the file, probably due to the broken javascript (the javascript tries to open an index via name, but index names aren't yet supported by mobiunpack).

I don't know what did went wrong for you, but at least the image handling works fine for the file.

Here is the log (some strings were repaced by XXX):

python mobiunpack.py XXX.mobi test
MobiUnpack 0.29
  Copyright (c) 2009 Charles M. Hannum <root@ihack.net>
  With Images Support and Other Additions by P. Durrant and K. Hendricks
  With Dictionary Support and Other Additions by S. Siebert
Unpacking Book...
Mobipocket version 6
Warning: Unknown metadata with id 404 found
Huffdic compression
Unpack raw html
Info: Document contains orthographic index, handle as dictionary
Info: Index doesn't contain entry length tags
Read dictionary index data
Warning: There are unprocessed index bytes left: XX XX
[...]
Warning: There are unprocessed index bytes left: XX XX
Decode images
Find link anchors
Insert data into html
Insert hrefs into html
Remove empty anchors from html
Insert image references into html
Write html
Write opf
Completed
The Mobi HTML Markup Language File can be found at: test\XXX.html

C:\mobidict>cd test

C:\mobidict\test>mobigen -c2 XXX.opf

*****************************************
* Mobipocket mobigen.exe V6.2 build 43  *
* A command line e-book compiler        *
* Copyright Mobipocket.com 2003-2008    *
*****************************************

opt compression: Mobipocket huffdic compression
opt version: try to minimize (default)
Info(prcgen): Added metadata dc:Title        "XXX"
Info(prcgen): Added metadata dc:Date         "XXX"
Info(prcgen): Added metadata dc:Creator      "XXX"
Info(prcgen): Added metadata dc:Publisher    "XXX"
Info(prcgen): Added metadata dc:Subject      "Dictionary"
Info(prcgen): Added metadata Short dic label "XXX"
Warning(prcgen): Guide title is empty. Item is ignored
Info(prcgen): Parsing files  0000001
Info(prcgen): Resolving hyperlinks
Info(prcgen): Resolving start reading location
Warning(prcgen): The start reading location could not be resolved.
Warning(prcgen): Cover is too small :   C:\mobidict\test\images\image00XXX.jpeg
Info(prcgen/inflections): Number of new <idx:infl> inflection rules:  0000XXX
Info(prcgen/inflections): Of which rules used only once or twice:     0000XXX
Info(prcgen/inflections): Number of inflection rule groups:           0000XXX
Info(prcgen): Computing UNICODE ranges used in the book
Info(prcgen): Found UNICODE range: Basic Latin [20..7E]
Info(prcgen): Found UNICODE range: Latin-1 Supplement [A0..FF]
Info(prcgen/compress): Compiling source text for compression (4096 passes max).
Pass  0000001
Info(prcgen/compress): Compiling source text for compression (4096 passes max).
Pass  0000002
Info(prcgen/compress): Compiling source text for compression (4096 passes max).
Pass  0000004
Info(prcgen/compress): Compiling source text for compression (4096 passes max).
Pass  0000008
Info(prcgen/compress): Compiling source text for compression (4096 passes max).
Pass  0000016
Info(prcgen/compress): Compiling source text for compression (4096 passes max).
Pass  0000032
Info(prcgen/compress): Compiling source text for compression (4096 passes max).
Pass  0000064
Info(prcgen/compress): Compiling source text for compression (4096 passes max).
Pass  0000128
Info(prcgen/compress): Compiling source text for compression (4096 passes max).
Pass  0000256
Info(prcgen/compress): Compiling source text for compression (4096 passes max).
Pass  0000512
Info(prcgen/compress): Compiling source text for compression (4096 passes max).
Pass  0001024
Info(prcgen/compress): Compiling source text for compression (4096 passes max).
Pass  0002048
Info(prcgen/compress): Compiling source text for compression (4096 passes max).
Pass  0002174
Info(prcgen/compress): Compression pass  0000001
Info(prcgen/compress): Compression pass  0000002
Info(prcgen/compress): Compression pass  0000003
Info(prcgen/compress): Text compressed to (in % of original size):  010.18%
Info(prcgen/compress): Compression dictionary statistics:  001974180 bytes 00018
3450 entries
Info(prcgen/compress): Compression pass  0000004
Info(prcgen/compress): Text compressed to (in % of original size):  009.54%
Info(prcgen/compress): Compression dictionary statistics:  001402428 bytes 00012
8447 entries
Info(prcgen/compress): Compression pass  0000005
Info(prcgen/compress): Text compressed to (in % of original size):  009.32%
Info(prcgen/compress): Compression dictionary statistics:  001118938 bytes 00010
1641 entries
Info(prcgen/compress): Compression pass  0000006
Info(prcgen/compress): Text compressed to (in % of original size):  009.12%
Info(prcgen/compress): Compression dictionary statistics:  000894372 bytes 00008
1123 entries
Info(prcgen/compress): Compression pass  0000007
Info(prcgen/compress): Text compressed to (in % of original size):  009.00%
Info(prcgen/compress): Compression dictionary statistics:  000721806 bytes 00006
4837 entries
Info(prcgen/compress): Compression pass  0000008
Info(prcgen/compress): Text compressed to (in % of original size):  008.93%
Info(prcgen/compress): Compression dictionary statistics:  000586074 bytes 00005
1866 entries
Info(prcgen/compress): Compression pass  0000009
Info(prcgen/compress): Text compressed to (in % of original size):  008.90%
Info(prcgen/compress): Compression dictionary statistics:  000475936 bytes 00004
1489 entries
Info(prcgen/compress): Compression pass  0000010
Info(prcgen/compress): Text compressed to (in % of original size):  008.90%
Info(prcgen/compress): Compression dictionary statistics:  000388794 bytes 00003
3203 entries
Info(prcgen/compress): Compression pass  0000011
Info(prcgen/compress): Text compressed to (in % of original size):  008.91%
Info(prcgen/compress): Compression dictionary statistics:  000314792 bytes 00002
6574 entries
Info(prcgen/compress): Advanced compression successful (decoded and verified).
Info(prcgen): Final stats - text compressed to (in % of original size):  008.91%

Info(prcgen): The document identifier is: "XXX"
Info(prcgen): The file format version is V6
Info(prcgen): Saving MOBI file
Info(prcgen): MOBI File generated with WARNINGS!

Ciao,
Steffen

[naisren]

I guess I used the old version of mobigen, which is similar with current mobi publisher version.
The following version could not repeat the warnings about images.

***********************************************
* Amazon.com kindlegen(Windows) V1.0 build 85 *
* A command line e-book compiler *
* Copyright Amazon.com 2009 *
***********************************************

Thanks for your quick test and your understanding (XXX replacement).

With the javascript part, I always think it is excellent for this reader software, which make the mobi reader more attractive. I checked many official mobi dics, old or new, almost every dic uses the javascript.

The javascript in mobi is not perfect yet, which is a good point when a professional ebook, such as medical one, needs simple table handling inquiry. Sooner or later Kindle would fell obliged to apply the javasript into different platforms readers.

[siebert]

Originally Posted by naisren

With the javascript part, I always think it is excellent for this reader software, which make the mobi reader more attractive. I checked many official mobi dics, old or new, almost every dic uses the javascript.

AFAIK a dictionary has to use JavaScript to implement the search form to locate a specific entry.

The javascript in mobi is not perfect yet, which is a good point when a professional ebook, such as medical one, needs simple table handling inquiry. Sooner or later Kindle would fell obliged to apply the javasript into different platforms readers.

I doubt that Amazon will port the JavaScript support for the Kindle and the Kindle apps, as for dictionary lookup within a book it's unnecessary and apart from dictionaries there seem to be very few books using JavaScript.

I'm just wondering, if you want to use the dictionary with Mobipocket Reader, what is your reason to unpack and repack it?

I created the dictionary support for mobunpack just because the dictionary I want to use in the Kindle app has a very wasteful formatting which means in most cases the content of the popup window doesn't contain the relevant information and I have to switch to the full dictionary display, which is rather inconvenient.

So I unpacked the dictionary, removed the javascript stuff and reformatted it and now most of the time the popup window content is sufficient.

Ciao,
Steffen

[naisren]

Originally Posted by siebert

AFAIK a dictionary has to use JavaScript to implement the search form to locate a specific entry.



I doubt that Amazon will port the JavaScript support for the Kindle and the Kindle apps, as for dictionary lookup within a book it's unnecessary and apart from dictionaries there seem to be very few books using JavaScript.

I'm just wondering, if you want to use the dictionary with Mobipocket Reader, what is your reason to unpack and repack it?

I created the dictionary support for mobunpack just because the dictionary I want to use in the Kindle app has a very wasteful formatting which means in most cases the content of the popup window doesn't contain the relevant information and I have to switch to the full dictionary display, which is rather inconvenient.
Steffen

Thanks for your views, and I try to use a picture to cobble my thoughts together to answer your questions here. I use WM phone a lot, I tried Kindle applications on different platforms, such as Kindle3, Android, IPad, I think WM mobipocket is good for me. The shot pic is from my TP2.
Anyway, I am satisfied with the current decoder tool, thanks everyone for developing this beautiful tool.

[karunaji]

Thank you for mobi unpacker. That is exactly what I need as I wanted to fix readability for my dictionary.

Unfortunately I got the message Error: Dictionary contains multiple inflection index sections, which is not yet supported so the recreated dictionary file do not contain inflection. Can it be implemented somehow?

[siebert]

Originally Posted by karunaji

Unfortunately I got the message Error: Dictionary contains multiple inflection index sections, which is not yet supported so the recreated dictionary file do not contain inflection. Can it be implemented somehow?

Of course it can, but I was just too lazy, as the dictionary I was interested in didn't need it.

Ciao,
Steffen

[pdurrant]

Originally Posted by karunaji

Thank you for mobi unpacker. That is exactly what I need as I wanted to fix readability for my dictionary.

Unfortunately I got the message Error: Dictionary contains multiple inflection index sections, which is not yet supported so the recreated dictionary file do not contain inflection. Can it be implemented somehow?

I have no interest in Mobipocket format dictionaries, so I won't be trying. Siebert is the one who reverse engineered the dictionary support. Although he claims to not be interested, you could try persuasion, or ask for help in implementing it yourself

[KevinH]

Hi,

FYI, I have attached a mobiunpack_v0.30.py in case anyone wants it. It has only slight changes from the official 0.29 version, all related to metadata.

Changes include:

Add <meta /> tags for:

- Cover ThumbNail Image
- Text to Speech Disabled Flag
- Font Signature encoded as hex
- Tamper Proof Keys encoded as hex
- All unknown metadata keys as hex strings

This allows someone to completely recreate any EXTH header region if they so needed and can help with further debugging/identification of unknown metadata keys.

[karunaji]

Originally Posted by siebert

Of course it can, but I was just too lazy, as the dictionary I was interested in didn't need it.

Ciao,
Steffen

Hi Steffen,

How hard it would be to figure this out? I tried to look at the format description but I probably won't have that much time to figure it out. I had some impression that it is not fully described.

I want it because I have bought PONS DE>EN dictionary and I use it to practice reading German texts. This dictionary provides word pronunciation in square brackets [] but in Kindle pop-up window it appears empty. I have to press ENTER and it's rather inconvenient. I decoded the dictionary and I can see that the pronunciation is composed of small images. But Kindle font actually contains extended IPA characters so it should be trivial to replace the images to chars and repack the dictionary.

I just tried to decode another dictionary DE>RU and it also contains multiple inflection index sections.

[siebert]

Originally Posted by karunaji

How hard it would be to figure this out? I tried to look at the format description but I probably won't have that much time to figure it out.

I think you'll learn the needed things better from the code than from the format description.

I don't know how fluent your Python is, but the multiple index section support shouldn't be very hard. The most important information you have to figure out is whether the additional sections use their own different tag tables or not. If they do, you have to hold multiple tag tables in addition to multiple sections, but that's also doable, just a bit more code.

You can run mobiunpack with WRITE_RAW_DATA set to True to dump the index sections as individual files and use a hex editor to analyse the data.

But I'm afraid that in your case (german-english dictionary) you will also face an issue with the inflection rules, as the current implementation doesn't handle words with special characters like german umlauts properly. That might be just a simple text encoding issue, but it could also be something which needs additional reverse engineering effort.

I want it because I have bought PONS DE>EN dictionary and I use it to practice reading German texts. This dictionary provides word pronunciation in square brackets [] but in Kindle pop-up window it appears empty. I have to press ENTER and it's rather inconvenient. I decoded the dictionary and I can see that the pronunciation is composed of small images. But Kindle font actually contains extended IPA characters so it should be trivial to replace the images to chars and repack the dictionary.

Do I understand that correct that on the Kindle device the popup window doesn't support images so you can't read the pronuncation?

In the Kindle app the pronuncation is displayed fine, but I had to reformat my dictionary because the pronuncation and other information takes so much space that the popup window doesn't contain the actual translation for most words. So I removed pronuncation and unnecessary whitespace from the formatting to get a usable dictionary for the Kindle app.

I just tried to decode another dictionary DE>RU and it also contains multiple inflection index sections.

The german language seem to require much more inflection rules than for example english. So I would assume that most if not all german to whatever dictionaries will contain multiple inflection index sections.

If you want to test more dictionaries, mobipocket.com provides free sample downloads for (all?) dictionaries they sell, the samples are also without DRM, so you can just run mobiunpack on them.

Ciao,
Steffen

[karunaji]

Thanks Steffen, for your detailed explanations.

I am not sure if I have enough patience to do this but I can always hope.

Originally Posted by siebert

But I'm afraid that in your case (german-english dictionary) you will also face an issue with the inflection rules, as the current implementation doesn't handle words with special characters like german umlauts properly. That might be just a simple text encoding issue, but it could also be something which needs additional reverse engineering effort.

I noticed that with another dictionary. I suspect that it is a coding issue. That dictionary was in UTF-8 but it was expanded like Windows-1252.

Do I understand that correct that on the Kindle device the popup window doesn't support images so you can't read the pronuncation?

It is exactly how it is.

[avid-e-reader]

Just found this thread and tool. Great stuff! Nice to see the HTML that kindlegen converts to! Helps to figure out what works and what doesn't.

However, I ran a little test book through kindlegen, then mobiunpack, and then kindlegen again, and somehow, the Kindle for PC won't display the cover page anymore.

I'll attach the .mobi that I unpacked, which should suffice for testing.

Also, a curiosity question: the Kindle claims to be able to play .mp3 files. Has anyone verified that, and does mobiunpack deal with such files? I have a background idea of adding some .mp3 to one of my ebooks, if it works. Kindle claims some video support too, but I'm not interested in that.

[pdurrant]

Originally Posted by avid-e-reader

Just found this thread and tool. Great stuff! Nice to see the HTML that kindlegen converts to! Helps to figure out what works and what doesn't.

However, I ran a little test book through kindlegen, then mobiunpack, and then kindlegen again, and somehow, the Kindle for PC won't display the cover page anymore.

I'll attach the .mobi that I unpacked, which should suffice for testing.

Testing on Mac with KindleGen 1.2 and Mobiunpack 0.29, it works fine for me. Copying the books to Windows XP, your sample book crashes my copy of Kindle for PC 1.5, but the recompiled version loads fine, and Go/Cover works too.

I think you need to update your copy of KindleGen.

[avid-e-reader]

I tried mentally comparing my original .opf with the generated one, and there was no entry in the manifest for the cover image file. After adding one, then it regenerates fine. Would that indicate a problem in the kindlegen? I guess there is a newer one... but seems to be an omission from the .opf -- unless the newer one doesn't need the manifest entry.