KindleUnpack (MobiUnpack): Extracts text, images and metadata from Kindle/Mobi files

[KevinH]

Hi DiapDealer and tkeo,

I pushed the fix to both master and python2and3 branches. It will be in the next release once I get some free time to incorporate tkeo's audio and video unpacking code into both the master and python2and3 branches.

tkeo if you want me to push a patch that will make using Furigana metadata under auto indicate a target of epub 3, please provide patches for both branches if possible.

DiapDealer, please tell the bug reporter thanks for reporting the bug!

Take care,

KevinH

[tkeo]

Hi Kevin,

I have made patches that make using Furigana metadata under auto indicate a target of epub 3.

But it cannot be tested in python2and3 branch because errors occur in command line mode as follows.

python2

Traceback (most recent call last):
  File "kindleunpack.py", line 11, in <module>
    from .compatibility_utils import PY2, binary_type, utf8_str, unicode_str
ValueError: Attempted relative import in non-package

python3

Traceback (most recent call last):
  File "kindleunpack.py", line 11, in <module>
    from .compatibility_utils import PY2, binary_type, utf8_str, unicode_str
SystemError: Parent module '' not loaded, cannot perform relative import

Thanks,

[PoP]

Catching up with the thread, I discovered that for some time KindleUnpack has experimentally been able to unpack dictionaries, and possibly can even unpack inflections!

Using the current release of KindleUnpack Calibre Plugin
I tried it successfully on a set of dictionaries Thank you developers.

But it would not work on my homebrew NADSAT dictionary and reported the following:

calibre, version 2.8.0
ERREUR : KindleUnpack - The Plugin v0.77.0: 2

Traceback (most recent call last):
  File "calibre_plugins.kindleunpack_plugin.extraction", line 186, in unpack_ebook
  File "calibre_plugins.kindleunpack_plugin.utilities", line 282, in unpackMOBI
  File "calibre_plugins.kindleunpack_plugin.core.kindleunpack", line 870, in unpackBook
  File "calibre_plugins.kindleunpack_plugin.core.kindleunpack", line 793, in process_all_mobi_headers
  File "calibre_plugins.kindleunpack_plugin.core.kindleunpack", line 581, in processMobi7
  File "calibre_plugins.kindleunpack_plugin.core.mobi_dict", line 204, in getPositionMap
KeyError: 2

Thought I would just bring it to your attention.

[KevinH]

Hi tkeo,

I recently merged a pull request that made those changes. I will checkout the python2and3 branch and fix it to work.

And thanks for your patches. I will incorporate them as well.

KevinH

Originally Posted by tkeo

Hi Kevin,

I have made patches that make using Furigana metadata under auto indicate a target of epub 3.

But it cannot be tested in python2and3 branch because errors occur in command line mode as follows.

python2

Traceback (most recent call last):
  File "kindleunpack.py", line 11, in <module>
    from .compatibility_utils import PY2, binary_type, utf8_str, unicode_str
ValueError: Attempted relative import in non-package

python3

Traceback (most recent call last):
  File "kindleunpack.py", line 11, in <module>
    from .compatibility_utils import PY2, binary_type, utf8_str, unicode_str
SystemError: Parent module '' not loaded, cannot perform relative import

Thanks,

[KevinH]

Hi Pop,

Thanks for that bug report. It says that a fileposition does not exist that should. I would really need a copy of the dictionary ( or a sample that exhibits the same issue ) so that I can figure out what might be funny.

So please post a url to the problem dictionary or if you would prefer you can privately send me a url via personal messaging on this site (pm KevinH) so that I can track down the error and get it fixed.

Thanks,

KevinH

Originally Posted by PoP

Catching up with the thread, I discovered that for some time KindleUnpack has experimentally been able to unpack dictionaries, and possibly can even unpack inflections!

Using the current release of KindleUnpack Calibre Plugin
I tried it successfully on a set of dictionaries Thank you developers.

But it would not work on my homebrew NADSAT dictionary and reported the following:

calibre, version 2.8.0
ERREUR : KindleUnpack - The Plugin v0.77.0: 2

Traceback (most recent call last):
  File "calibre_plugins.kindleunpack_plugin.extraction", line 186, in unpack_ebook
  File "calibre_plugins.kindleunpack_plugin.utilities", line 282, in unpackMOBI
  File "calibre_plugins.kindleunpack_plugin.core.kindleunpack", line 870, in unpackBook
  File "calibre_plugins.kindleunpack_plugin.core.kindleunpack", line 793, in process_all_mobi_headers
  File "calibre_plugins.kindleunpack_plugin.core.kindleunpack", line 581, in processMobi7
  File "calibre_plugins.kindleunpack_plugin.core.mobi_dict", line 204, in getPositionMap
KeyError: 2

Thought I would just bring it to your attention.

[DiapDealer]

Originally Posted by KevinH

Hi tkeo,

I recently merged a pull request that made those changes. I will checkout the python2and3 branch and fix it to work.

Oops. The relative imports thing is my fault. I never tested running from the command-line ... only from the gui "package." Ironic, since I was looking for a way to simplify importing the command-line scripts into other projects.

In light of this, it's probably best to just roll back that last merge entirely. Even separating the gui-helpers from the core unpacking scripts is probably not all that important at this point (and we can easily add that back in later if desired).

[KevinH]

Hi DiapDealer,

No need to rollback as I was able to get both to work by adding:

__path__ = ["lib", ".", "kindleunpack"]

at the top to force it to be a package and accept either lib.blah or .blah as import paths to allow both gui and command line

So no worries. python2and3 via the command line should now work again.

KevinH

[PoP]

Originally Posted by KevinH

Hi Pop,

...
I would really need a copy of the dictionary ( or a sample that exhibits the same issue ) so that I can figure out what might be funny.

So please post a url to the problem dictionary
...

Thanks KevinH.

The dictionary is in the last zip attached to this post :https://www.mobileread.com/forums/sho...d.php?t=205634

[KevinH]

Hi tkeo,

I merged your fixes to target epub3 when Furigana metadata is present. I took the liberty of modifying your python2and3 patch to remove the .keys() since simply testing in metadata will work well with both python2 and python3.

Thanks!

KevinH

[DiapDealer]

Originally Posted by KevinH

Hi DiapDealer,

No need to rollback as I was able to get both to work by adding:

__path__ = ["lib", ".", "kindleunpack"]

at the top to force it to be a package and accept either lib.blah or .blah as import paths to allow both gui and command line

So no worries. python2and3 via the command line should now work again.

KevinH

Thanks for fixing that Kevin!
Was able to sync my fork with your master to pick up that fix, and will begin looking at converting the prefs store from ini to json.

[KevinH]

Hi PoP,

The error meant that one entry was missing an end position (it did not exist in the tagMap).

I think this is caused by your very very last entry in the dictionary not having a proper ending </idx:entry> tag.

Here is the snippet of code:

<idx:entry name="word" scriptable="yes">
        <h2>
          <idx:orth>Zvook</idx:orth><idx:key key="Zvook">
        </h2>
        <I>fr:</I> son <I>en:</I> sound <I>ru:</I> звук, zvuk (sound)

<div> <img hspace="0" vspace="0" align="middle" src="../Images/sep.gif"/> <table width="100%" bgcolor="#992211"><tr><th widht="100%" height="2px"></th></tr></table> </di\
v>

    </mbp:frameset>


  </body>
</html>

Notice there is no </idx:entry> to match the opening <idx:entry blah> tag for this very last word.

So please try adding the final </idx:entry> tag right before the </mbp:frameset> and recompiling.

It should then unpack successfully (I hope!). Please let me know if that does not do the trick.

Also, I just noticed that your use of <idx:key> does not seem to have an ending tag either (please check this all throughout your dictionary). If there is no <idx:key> content then it should be self-closed:
<idx:key name="blah" /> otherwise there should be a matching ending </idx:key> tag someplace as well for each beginning tag.

Take care,

KevinH

Originally Posted by PoP

Thanks KevinH.

The dictionary is in the last zip attached to this post :https://www.mobileread.com/forums/sho...d.php?t=205634

[PoP]

Originally Posted by KevinH

Hi PoP,

The error meant that one entry was missing an end position (it did not exist in the tagMap).

I think this is caused by your very very last entry in the dictionary not having a proper ending </idx:entry> tag. ... So please try adding the final </idx:entry> tag

It should then unpack successfully (I hope!). Please let me know if that does not do the trick.

Also, I just noticed that your use of <idx:key> does not seem to have an ending tag either (please check this all throughout your dictionary). If there is no <idx:key> content then it should be self-closed:
<idx:key name="blah" /> otherwise there should be a matching ending </idx:key> tag someplace as well for each beginning tag.

Take care,

KevinH

Thanks: adding the missing idx:entry closing tag for the last dictionary entry allowed KindleUnpack to work flawlessly. I am impressed, all the index entries, including the inflections, are properly extracted. Great job.

I also more properly self-closed all the idx:key tags.

As usual, find the updated dictionary here.

[tkeo]

Hi Kevin,

I have made two patches for the python2and3 branch.

1. to unpack AUDI and VIDE sections.

As same as the patch posted previously,
the AUDI and VIDE sections are unpacked into HDimages.
Tags in xhtmls are not linked to unpacked files.
It need to change unpacking directory and to make the tags linked.

2. to use OrderedDict in DEBUG mode.

Please commit these patches if acceptable.

Take care,
tkeo

[KevinH]

Hi tkeo,

The audio video one is broken. According to Amazon publishing guidelines only Mobi 7 support Audio/Video and any audio/video files must be stored in a folder named "audiovideo".

So hold off on that. It was never meant to work in the KF8 part. At least not yet, which is why the link/urls were not updated or replaced by proper internal Kindle ones in the KF8 part.

BTW: I spent some time looking at audio/video and it seems that in the mobi 7 header, offset 0xe8 which we now call "unknown3" always points to the first record where extra indexes are stored. These extra INDX sections have yet to be decoded. They are related to the audio and video sections and links. Furthermore offset 0xc4 in the mobi 7 header (we refer to this as "unknown0") now seems to have a 1.

So we really need to figure out what these extra INDX sections are and how to decode them, what/why the "unknown3" points to them, etc.

So there is still too much reverse-engineering to be done before we can support audio/video in the older Mobi 7's. And when we do understand them, the unpacking part is easy and should go into a new folder called "audiovideo" inside the mobi7 folder when unpacked.

Take care,

KevinH

ps. I have committed your DEBUG use of OrderedDict patch. Thank you!

[elchamaco]

Hi, KevinH,

I'm trying to decompress a dictionary generated with kindlegen, the dictionary have inflected forms but kindleunpack didn't notice them. There is a log from kindlegen and i see what's the problem, they put the inflected forms as new definitions but in the same definition, the dictionary works but kindleunpacks fails to decompress.

They use somthing like this:

<idx:entry>
<idx:orth value="palabra"></idx:orth>
<idx:orth value="inflectedform1"></idx:orth>
...
<idx:orth value="inflectedform2"></idx:orth>
</idx:entry>
definición

I found in kindlegenlog part of a definition so i think that's the problem:

eu"><a id="sorraríeu"></a></idx:orth>
<idx:orth value="sorrat"><a id="sorrat"></a></idx:orth>
<idx:orth value="sorrats"><a id="sorrats"></a></idx:orth>
<idx:orth value="sorrava"><a id="sorrava"></a></idx:orth>
<idx:orth value="sorraven"><a id="sorraven"></a></idx:orth>
<idx:orth value="sorraves"><a id="sorraves"></a></idx:orth>
<idx:orth value="sorrem"><a id="sorrem"></a></idx:orth>
<idx:orth value="sorren"><a id="sorren"></a></idx:orth>
<idx:orth value="sorres"><a id="sorres"></a></idx:orth>
<idx:orth value="sorressin"><a id="sorressin"></a></idx:orth>
<idx:orth value="sorressis"><a id="sorressis"></a></idx:orth>
<idx:orth value="sorreu"><a id="sorreu"></a></idx:orth>
<idx:orth value="sorri"><a id="sorri"></a></idx:orth>
<idx:orth value="sorrin"><a id="sorrin"></a></idx:orth>
<idx:orth value="sorris"><a id="sorris"></a></idx:orth>
<idx:orth value="sorro"><a id="sorro"></a></idx:orth>
<idx:orth value="sorrà"><a id="sorrà"></a></idx:orth>
<idx:orth value="sorràrem"><a id="sorràrem"></a></idx:orth>
<idx:orth value="sorràreu"><a id="sorràreu"></a></idx:orth>
<idx:orth value="sorràvem"><a id="sorràvem"></a></idx:orth>
<idx:orth value="sorràveu"><a id="sorràveu"></a></idx:orth>
<idx:orth value="sorrés"><a id="sorrés"></a></idx:orth>
<idx:orth value="sorréssim"><a id="sorréssim"></a></idx:orth>
<idx:orth value="sorréssiu"><a id="sorréssiu"></a></idx:orth>
<idx:orth value="sorrí"><a id="sorrí"></a></idx:orth>
<p class="lema"><span class="navy"><sup>■</sup></span><span class="navy"><strong>sorrar</strong></span></p><p class="dict"><a href="="=

When they finish with all the orth's that in fact are the inflected forms then they begin with the definiction. In fact they don't finish the idx:entry before the definition, but kindleunpack fails to handle and puts the close of the idx:entry before the definition, i found more dictionaries with that problem, i thought it was a problem with the dictionary but now with this log i think it's a problem with kindleunpack that puts the close tag in a bad place before the definition.

It's possible to handle this kind of dictionaries and unpack all the terms? I can provide the dictionary.

Regards