KindleUnpack (MobiUnpack): Extracts text, images and metadata from Kindle/Mobi files

[kovidgoyal]

Ah, you want to only allow editing of the content, not the TOC...then yes, decompiling the NCX from the INDX records is fine.

[siebert]

Originally Posted by fandrieu

Absolutely, btw does anyone knows a way to spot calibre-generated books / identify the book generator ?

The EXTH value 204 (creator software) is set by the official tools (kindlegen, mobigen, mobipocket creator) with distinct values, but unfortunately the latest calibre versions also set this value, pretending to be a kindlegen.

I asked Kovid to remove that "feature", but he refused to do so.

So the only way to find out if a mobi files was created by calibre or not is to search the contributor EXTH (value 108) for the string "calibre".

Ciao,
Steffen

[DiapDealer]

Hey fandrieu,

I can't find where it's happening for the life of me, but something in your code is doubling the html output. It doesn't happen with books that have no ncx, but if it does have ncx data, the html file being produced is the original html x 2: as in the whole book and then the whole book all over again.

Maybe you'll have better luck spotting it.

[fandrieu]

@kovid: thanks very much for your great app !

@sieberd: thanks for the info, i'll look into it.

Originally Posted by DiapDealer

I can't find where it's happening for the life of me, but something in your code is doubling the html output.

I found a file to debug that
Here's a fixed version including the typos mentioned earlier.

...

The bug was a weird collateral of the "ncx filepos anchor injection" code.
The duplication occured because an entry had -1 as filepos, meaning an extra large "dataList".

The cause was IDXT parsing, some files have an extra null entry at the end.

I added two checks to prevent both problems.

...

REUP: after posting that i realized it'd be better to use header['count'] to determine when to stop parsing IDXT...

[kaizoku]

I am getting these errors with mobi:

Unpacking Book...
Mobipocket version 6
Warning: Unknown metadata with id 405 found
Warning: Unknown metadata with id 406 found
Warning: Unknown metadata with id 407 found
Warning: Unknown metadata with id 403 found
Write ncx
Palmdoc compression
Unpack raw html
Decode images
Find link anchors
Insert data into html
Insert hrefs into html
Remove empty anchors from html
Insert image references into html
Write html
Write opf
Completed

But no dat was created just ncx opf and html

[pdurrant]

Originally Posted by kaizoku

I am getting these errors with mobi:

Unpacking Book...
Mobipocket version 6
Warning: Unknown metadata with id 405 found
Warning: Unknown metadata with id 406 found
Warning: Unknown metadata with id 407 found
Warning: Unknown metadata with id 403 found

[snip]

But no dat was created just ncx opf and html

Those aren't errors, just warnings. The data in those EXTH sections will be in the opf inside some comments, as far as I remember.

[Doitsu]

While the tool works great with regular books, it seems to have problems with some dictionaries. When I unpacked my simple Swedish-English dictionary, which I uploaded some time ago, I noticed that the <DictionaryInLanguage> value was not correctly recovered. My original .opf file contained the following entries:

<DictionaryInLanguage>sv</DictionaryInLanguage>
<DictionaryOutLanguage>en-us</DictionaryOutLanguage>

However, the reverse engineered .opf file contained the following entries:

<DictionaryInLanguage>en</DictionaryInLanguage>
<DictionaryOutLanguage>en-us</DictionaryOutLanguage>

Since the <DictionaryInLanguage> value is used by the Kindle for automatic dictionary selection, a wrong value will cause problems.

I also got a lot of "Delete operation of inflection rule failed" error messages, but I remember having read somewhere that there are still problems with inflections.

[pdurrant]

Originally Posted by Doitsu

While the tool works great with regular books, it seems to have problems with some dictionaries. When I unpacked my simple Swedish-English dictionary, which I uploaded some time ago, I noticed that the <DictionaryInLanguage> value was not correctly recovered. My original .opf file contained the following entries:

<DictionaryInLanguage>sv</DictionaryInLanguage>
<DictionaryOutLanguage>en-us</DictionaryOutLanguage>

However, the reverse engineered .opf file contained the following entries:

<DictionaryInLanguage>en</DictionaryInLanguage>
<DictionaryOutLanguage>en-us</DictionaryOutLanguage>

Since the <DictionaryInLanguage> value is used by the Kindle for automatic dictionary selection, a wrong value will cause problems.

I also got a lot of "Delete operation of inflection rule failed" error messages, but I remember having read somewhere that there are still problems with inflections.

Thanks for the bug report. Hopefully the guys working on the dictionary support can fix it up.

[pdurrant]

Originally Posted by pdurrant

Thanks for the bug report. Hopefully the guys working on the dictionary support can fix it up.

Well, actually I took a quick look, and added a language entry to fix that problem. There's still some work to me done on decoding the inflection rules. I'm sure that "Error: Delete operation of inflection rule failed" needs to be fixed. Perhaps someone with the source to a dictionary with infections could have a go?

Anyway, I've uploaded version 0.32 to the fifth post in this thread. It includes some refactoring by DiapDealer, which will hopefully make maintenance easier.

[siebert]

Originally Posted by pdurrant

There's still some work to me done on decoding the inflection rules. I'm sure that "Error: Delete operation of inflection rule failed" needs to be fixed. Perhaps someone with the source to a dictionary with infections could have a go?

The delete rule contains the letter which should be deleted. The error is given if that letter doesn't match the current letter of the word at the position where the deletion should be performed.

Normally this shouldn't happen, but for non-ascii letters the letters don't match. I assume that the text encodings of the rules and the actual text are different. But I have no idea what encoding is used in the rules.

I could provide the source of a minimal dictionary which shows the error if needed.

Ciao,
Steffen

[pdurrant]

Originally Posted by siebert

I could provide the source of a minimal dictionary which shows the error if needed.

If you could do that (source and compiled dictionary, if possible), I'll take a look. Thanks.

[siebert]

Ok, here is the promised sample dictionary. The first entry creates the error.

deleteruleerror.zip

Ciao,
Steffen

[pdurrant]

Originally Posted by siebert

Ok, here is the promised sample dictionary. The first entry creates the error.

In one of the places in the sample dictionary, the german double-s ß is stored at 0x0573 instead of as 0xDF (the Windows Latin-1 encoding for ß).

A quick hack at the right place to substitute 0xDF back in for 0x0573 fixes things for this instance.

Unfortunately, I don't really understand why the error is happening, and it isn't a general fix — there are still problems with the swedish dictionary mentioned above.

Perhaps with access to the source for the swedish dictionary, it might be possible to work out what's going on.

[Doitsu]

Originally Posted by pdurrant

Perhaps with access to the source for the swedish dictionary, it might be possible to work out what's going on.

I did some tests and found out that the script seems to stumble over inflection entries with both a hyphen and an umlaut in them. For example:

<idx:infl><idx:iform value="abc-böckers"/></idx:infl>

Please find attached a small sample of the Swedish dictionary whose first entry will cause 4 Error: Delete operation of inflection rule failed messages when the .prc file is unpacked.

Unfortunately, there seem to be other serious issues with accented characters which you'll see when you look at the original and the reconstructed .html files.

Even though the reconstructed dictionary looks the same as the original when it's compiled, it no longer works as a dictionary.

[siebert]

Originally Posted by Doitsu

Even though the reconstructed dictionary looks the same as the original when it's compiled, it no longer works as a dictionary.

What exactly do you mean with "no longer works as a dictionary"?

In mobipocket reader, a dictionary uses javascript to implement dictionary search. This might indeed not work.

My focus was to use the recompiled dictionary in the kindle app (as the formatting of my originial dictionary made it unsuitable for the popup dictionary window), which should work as the kindle app only uses the dicitionary index and doesn't support javascript (so I removed the javascript code before recompiling the dictionary).

Ciao,
Steffen