KindleUnpack (MobiUnpack): Extracts text, images and metadata from Kindle/Mobi files

[eureka]

Originally Posted by KevinH

Yes that would be a good idea but I had not figured out how to push the historic release tags from my local tree to github yet, so there were no releases to link to when I made the post. I only found out how to do that today.

So I just was telling you what you already knew and was planning to use. Sorry for noise. Nice work with tagging!

[quiris]

Originally Posted by quiris

Is there an open github.com repository with KindleUnpack source code? It would be easier to track changes…

Originally Posted by KevinH

The url for the repository is:
https://github.com/kevinhendricks/KindleUnpack

At last… Thank you

[eureka]

Originally Posted by eschwartz

Yep, basically you get a pretty icon and a clicky to your GitHub profile in the commit log plus credit here: https://github.com/kevinhendricks/Ki...kevinhendricks

And credit in contributors list.

[Peter Ahlstrom]

Originally Posted by KevinH

Thank you so much for submitting your bug report and test case as it helps us to better understand the inner workings of the mobi format!

Great! Glad I was able to help.

[tkeo]

Hi Kevin,

I have fixed a bug in the python2and3 branch on the github.
In addition, I have modified to make eaier to compare the unpacked results by python 2 and python 3.

Take care,

[KevinH]

Hi tkeo,
Do we really need to overwrite all dictionaries with ordered dictionaries?

I would prefer we explicitly use ordered dicts only where needed (and none are actually truly needed here ... ). If you have some justification as to why you need an ordered dict then import it as something other than dict and use it only where needed explicitly.

Also, the flake8 style checks require a space after the "#" for comments. Better yet simply delete those extra HtmlParser related lines since the compatibility_utils unescapeit removes their use completely.

If you can fix both of these and have a github account, please create a fork and pull request, so that you get the proper credit in the history.

Thanks,

Kevin

[tkeo]

Hi Kevin,

Originally Posted by KevinH

I would prefer we explicitly use ordered dicts only where needed (and none are actually truly needed here ... ). If you have some justification as to why you need an ordered dict then import it as something other than dict and use it only where needed explicitly.

I compare the unpacking results of two versions by a diff command or a GUI diff tool. Using the ordered dictionaries makes testing easier greatly. I have found the necessary dictionaryies to be needed keeping orders are metadata in mobi_header and tattr in mobi_k8resc.py for my demand.
So, could you change dict to OrderedDict if acceptable for you?

BTW, are there any case that keeping the oders of data in a mobi helps to understand the functionality of kindlegen better?

If you can fix both of these and have a github account, please create a fork and pull request, so that you get the proper credit in the history.

I have made a new patch; but, I do not have any github account. Could you update it on my behalf?

Thanks,

[KevinH]

Hi tkeo,

Do you understand that no xml parser guarantees order of attributes? The order of attributes simply does not matter when recreating the opf.

As for dumping the header, both DumpMobiHeader.py and the dump code in unpack will show you the order of the header values, the exth, and all of the sections themselves if you enable debug.

So neither of your use cases makes any sense.

Using an ordered dict to maintain order when no xml parser needs it just hurts execution time meaninglessly. An ordered dict actually requires a normal dict plus a doubly linked list, plus a second ordered dict to quickly find elements in the doubly linked list. All of that in unneccessary overhead for something that does not matter to any xml processor.

So I will not integrate an ordered dict patch unless you can explain why it is truly needed. If it is only so you can diff things, then incorporate its use into a DEBUG setting to enable it only when DEBUG is true, otherwise use normal dictionaries.

Hope this better explains things.

KevinH

ps. I have committed your bug fix for removal of HTMLParser

Originally Posted by tkeo

Hi Kevin,



I compare the unpacking results of two versions by a diff command or a GUI diff tool. Using the ordered dictionaries makes testing easier greatly. I have found the necessary dictionaryies to be needed keeping orders are metadata in mobi_header and tattr in mobi_k8resc.py for my demand.
So, could you change dict to OrderedDict if acceptable for you?

BTW, are there any case that keeping the oders of data in a mobi helps to understand the functionality of kindlegen better?


I have made a new patch; but, I do not have any github account. Could you update it on my behalf?

Thanks,

[tkeo]

Hi Kevin,

Originally Posted by KevinH

So I will not integrate an ordered dict patch unless you can explain why it is truly needed. If it is only so you can diff things, then incorporate its use into a DEBUG setting to enable it only when DEBUG is true, otherwise use normal dictionaries.

It is needed only for testing, not for structures of unpacked contents. So, enabling ordered dictionary in DEBUG mode is enough.

Thanks,

[DiapDealer]

@tkeo and/or KevinH:

Got an error report over in the plugin thread that I could use more eyes on. I'm not up to speed on all the newer epub3 processing code and this appears to have something to do with a KeyError in the processRefinesMetadata portion of mobi_opf.py.

Thanks.

https://www.mobileread.com/forums/sho...&postcount=281

[KevinH]

Hi DiapDealer,

Yes, this is a bug. It only happens using the Furigana related Exth values that are only generated by Japanese versions of kindlegen. The bug in this case only happens when the book is an epub 2 book, and not an epub 3. I would guess we have never tested this case before ;-).

We should check if the key is in the metadata under epub 2 just like we do when it is an epub3.

Here is the original problem code in mobi_opf.py (starting at line 605)

        if create_refines_metadata:
            for [EXTH, id, attrib, defaultid] in refines_list:
                if self.target_epubver == '3':
                    for i, value in id.items():
                        attrib[i] = ' id="%s"' % value

                    if EXTH in metadata.keys():
                        if len(metadata[EXTH]) == 1 and len(id) == 1:
                            self.createMetaTag(self.exth_solved_refines_metadata, 'file-as', metadata[EXTH][0], id[0])
                        else:
                            for i, value in enumerate(metadata[EXTH]):
                                self.createMetaTag(self.exth_refines_metadata, 'file-as', value, id.get(i, defaultid))
                else:
                    if len(metadata[EXTH]) == 1 and len(id) == 1:
                        attr = ' opf:file-as="%s"' % metadata[EXTH][0]
                        attrib[0] = attr
                    else:
                        for i, value in enumerate(metadata[EXTH]):
                            attr = ' id="#%s" opf:file-as="%s"\n' % (id.get(i, defaultid), value)
                            self.extra_attributes.append(attr)

It should instead look like the following to at least prevent the KeyError (in other words - to make the else clause mirror the if-then when it comes to Key checking:

        if create_refines_metadata:
            for [EXTH, id, attrib, defaultid] in refines_list:
                if self.target_epubver == '3':
                    for i, value in id.items():
                        attrib[i] = ' id="%s"' % value

                    if EXTH in metadata.keys():
                        if len(metadata[EXTH]) == 1 and len(id) == 1:
                            self.createMetaTag(self.exth_solved_refines_metadata, 'file-as', metadata[EXTH][0], id[0])
                        else:
                            for i, value in enumerate(metadata[EXTH]):
                                self.createMetaTag(self.exth_refines_metadata, 'file-as', value, id.get(i, defaultid))
                else:
                    if EXTH in metadata.keys():
                        if len(metadata[EXTH]) == 1 and len(id) == 1:
                            attr = ' opf:file-as="%s"' % metadata[EXTH][0]
                            attrib[0] = attr
                        else:
                            for i, value in enumerate(metadata[EXTH]):
                                attr = ' id="#%s" opf:file-as="%s"\n' % (id.get(i, defaultid), value)
                                self.extra_attributes.append(attr)

Please ask the reporter to give that change a try. If it works, I will commit it to the tree and its equivalent to the python2and3 branch as well.

Hope this does the trick.

KevinH

[DiapDealer]

Thanks Kevin. I'm going to try to get a test version of the plugin (with those changes) to the reporter sometime later today.

[tkeo]

Hi Kevin,

Thank you for your quick support.
Yes. It is a bug. I think have made it.


Correction.

Originally Posted by KevinH

It only happens using the Furigana related Exth values that are only generated by Japanese versions of kindlegen.

the Furigana related Exth values that are only generated by Japanese versions of KDP.

Thanks,

[tkeo]

Hi,

In "auto" mode, an ebook should be converted to epub3 if Furigana relating EXTH exist in it since it requires "refines" attributes.

I will modify mobi_opf.py to do so, after the bug fix is commited.

Thanks,

[DiapDealer]

Originally Posted by KevinH

Please ask the reporter to give that change a try. If it works, I will commit it to the tree and its equivalent to the python2and3 branch as well.

Hope this does the trick.

KevinH

That indeed, seemed to work for the azw3 the user was having trouble with. Thanks.