KindleUnpack (MobiUnpack): Extracts text, images and metadata from Kindle/Mobi files

[KevinH]

Hi,
Please post the full debug kindleunpack log someplace first so I can see what might be going on. It will be large so zip it up first before posting it as an attachment. If I can't see an obvious problem, I would be happy to examine the full dictionary and try to figure out how to decode it in kindleunpack if possible.

KevinH

[elchamaco]

Here it's the compressed log:

http://www66.zippyshare.com/v/63012785/file.html

Thanks

[KevinH]

Hi elchamaco,

Please try again but this time:

1. use very latest version KindleUnpack v0.77 which is available at this link:

https://www.mobileread.com/forums/sho...postcount=1074

2. Do NOT create a rar. Instead create a simple zip file as I asked

3. Remember to use the -d option in KindleUnpack (debug) to dump more information.

4. Do NOT upload it to a horrible/dodgy website like zippyshare!

Simply attach the zip file to your post here as an attachment.

The kindlegen build log seems to indicate that the dictionary is not correct when built. Error messages like that are not normal. If that is the case, I won't be able to help.

KevinH

Originally Posted by elchamaco

Here it's the compressed log:

http://www66.zippyshare.com/v/63012785/file.html

Thanks

[elchamaco]

Hi,

I generated the log with 0.77. The unpack gives the same result as 0.75.

Regards.

[KevinH]

Hi elchamaco,

Yes this is very different from all other dictionaries I have seen.

Examining the mobi7 header shows that there is no meta inflexion index
at all but a new index called index_names does seem to exist and be used.

Here is a snippet from the Mobi7 header. See the metainfindex field has been set to missing 0xFFFFFFFF but something called index_names does have a non 0xFFFFFFFF value and it points to a set of unknown indexes further along than the metaorthindex.

Dumping section 0, Mobipocket Header version: 7, total length 272
Mobipocket header from section 0
     Offset  Value Hex Dec        Description
0x000 (  0)     0x0002          2 compression_type
0x002 (  2)     0x0000          0 fill0
0x004 (  4) 0x054DEC84   88992900 text_length
0x008 (  8)     0x54DF      21727 text_records
0x00A ( 10)     0x1000       4096 max_section_size
0x00C ( 12)     0x0000          0 crypto_type
0x00E ( 14)     0x0000          0 fill1
0x010 ( 16)       MOBI            magic
0x014 ( 20) 0x00000100        256 header_length (from MOBI)
0x018 ( 24) 0x00000002          2 type
0x01C ( 28) 0x0000FDE9      65001 codepage
0x020 ( 32) 0x434E1E13 1129192979 unique_id
0x024 ( 36) 0x00000007          7 version
0x028 ( 40) 0x000054E1      21729 metaorthindex
0x02C ( 44) 0xFFFFFFFF 4294967295 metainflindex
0x030 ( 48) 0x000055E7      21991 index_names
0x034 ( 52) 0xFFFFFFFF 4294967295 index_keys

Here are snippets from the map of sections in the mobi ebook:

The metaorthindex points here:

21729 54E1  0x1C9B030 0x01F04    43458       0 Unknown INDX section, extracting as Unknown21729_INDX.dat
21730 54E2  0x1C9CF34 0x0FBF0    43460       0 Unknown INDX section, extracting as Unknown21730_INDX.dat
21731 54E3  0x1CACB24 0x0FBEC    43462       0 Unknown INDX section, extracting as Unknown21731_INDX.dat
...

--snip--

The index_names point here.

21991 55E7  0x2C72C3C 0x000E0    43982       0 Unknown INDX section, extracting as Unknown21991_INDX.dat
21992 55E8  0x2C72D1C 0x000CC    43984       0 Unknown INDX section,

So the current dictionary code will not deal with this at all as it never even looks at the "index_names" nor does it know how to interpret its data.

I think that field name was based on information in the Wiki here and there may be no one who remembers why it was named "index_names" as I did not name it.

So supporting these strange dictionaries which have errors in their own kindlegen logs and who use who use no inflections index may require a big reverse engineering effort.

This is not something I can take on soon. But if you post the dictionary someplace that is not so swamped by unknown attacks (say www.datafilehost.com where I at least know how to avoid the issues) and send a personal mail on this site to "KevinH" with the link. I will take a look at eventually supporting it, just not right now as I am tied up with Sigil projects.

Thanks,

KevinH

Originally Posted by elchamaco

Hi,

I generated the log with 0.77. The unpack gives the same result as 0.75.

Regards.

[elchamaco]

Done.

I've also uploaded a dictionary i've created in the same way the did. The dictionary works, kindleupack fails to decompress de inflected forms. I included the source files and the test dictionary compiled with mobipocket and with kindlegen, in both cases the result is the same.

Regards.

[KevinH]

Hi,
Here is a status update.

I looked at your dictionary and the index_names is meaningless. There is an index header but no real index data there. I am not sure why this empty index is even created.

So I went back and looked at the orthIndex data and *all* of the words are there but some are given a different tagMap values that point to the entry they are subelements of. The current code only unpacks main elements not subelements since they should have been inflections.

The problem is that both main elements and subelements are in sorted alphabetical order so some subelements actually come before the main element they are listed under. That means our current one pass algorithm to build up a dictionary will no longer work. So this is going to take some effort.

These types of dictionaries may "work" but they are incorrect and all of these subelements should simply be inflections which would make the dictionary much smaller in size when compiled into a mobi.

And, I really hate to slow down unpacking of all dictionaries just to accommodate these incorrectly amd inefficiently created dictionaries. So I am not sure I will add support for these.

If I do decide to modify KindleUnpack to support them, I will make the code convert all of these new subelements to inflections.

KevinH

Originally Posted by elchamaco

Done.

I've also uploaded a dictionary i've created in the same way the did. The dictionary works, kindleupack fails to decompress de inflected forms. I included the source files and the test dictionary compiled with mobipocket and with kindlegen, in both cases the result is the same.

Regards.

[elchamaco]

Thanks for looking it. Yes for sure they use an incorrect way of doing it, i've not seeing any document explaining to code the entries in that way.

It's very strange, and it's a commercial dictionary.

To not slowdown, using an algorithm to standard dictionaries and other when a dictionary uses this kind of inflected forms?

Thanks anyway taking time to view it.

[DiapDealer]

@tkeo (and/or KevinH)

I have a bug report in the KindleUnpack plugin thread that looks to be related to your ePub3/Furigana-related code. Any chance you could take a look?

[KevinH]

Hi DiapDealer,

We released v077 on Oct 17 or so and this bug was fixed on Oct. 30th or so. It is so rare we have not made a new release for just that fix.

So your best bet is to simply update to today's master or grab that Oct 30th commit (skip the part where an entire emacs backup file with ~ gets added to the tree!) ;-)

Take care,

KevinH

Originally Posted by DiapDealer

@tkeo (and/or KevinH)

I have a bug report in the KindleUnpack plugin thread that looks to be related to your ePub3/Furigana-related code. Any chance you could take a look?

[DiapDealer]

OK, cool. Thanks!

[KevinH]

Hi All,

The very latest KindleUnpack version v0.80 has now been released.

Please see:

https://github.com/kevinhendricks/KindleUnpack/releases

This version now includes support for running under both Python 2.7 and 3.4 and it has all of the latest bug fixes including a fix for the missing TOC links issue (thank you Kovid and DiapDealer!) and a fix for Japanese specific metadata entries generated by Amazon.


@Developers

I ran into many many conflict problems trying to merge/sync the python2and3 branch directly into master and basically gave up. Instead, I have moved the original master branch to oldmaster (to keep its full history) and then copied the python2and3 branch to become a new master.

The branches currently labelled "python2and3" and "oldmaster" are now obsolete.

The branch "master" should be used as the target for any changes/pull requests or patches. Also any proposed changes need to be tested under both python 2.7.X and python 3.4.X.

If you have previously checked out a tree from this repo, you will probably have to make a clean clone of the repository since a proper merge was never fully done.

Thanks,

KevinH

[odamizu]

Thank you KevinH, DiapDealer and everyone else who has contributed time and effort to this AWESOME tool!

[elmimmo]

When KindleUnpack unpacks a FXL KF8 and repackages it as a FXL EPUB 3, each XHTML document in the spine should have a `meta` element inside its `head` block describing the size of the viewport, in order for the FXL EPUB to be rendered correctly in compatible EPUB ereaders (iBooks, Google Play Books, Kobo…):

<meta name="viewport" content="width=575, height=749"/>

Besides, the default EPUB version KindleUnpack repackages FXL KF8 as should always be EPUB 3 IMHO (currently defaults to 2), since such is the version that has a specified FXL variant.

Should I add these as issues at KindleUnpack's Github repo?

[KevinH]

Please post links to a completely valid fixed layout KF8 made via kindlegen (no drm) and the valid fixed layout epub3 that was used as input to kindlegen and I will work out how to properly unpack it and make the needed changes to KindleUnpack.

Thanks,

KevinH