New dictionary format of firmware 2.14

[mnjkl]

Does any one know what is the format of dictionary in firmware 2.14/2.15.
How to decode and encode it? Can we replace it with our dictionary?

1 the file name is '.a '
2 each file header start with 'NJDC'

[PeterT]

I don't think those are dictionaries. Take a look in .kobo/dict when the device is attached. You will see the dictionaries you have installed. They appear to be zip files, with the contents of them being a series of html files and one called "words". Words looks to have a signature of "We love Marisa.".

[tshering]

Unfortunately, the html files are not in plain text. Up to 2.0.0 or so they were gz compressed. Since then they seem to be encrypted, I guess because of copyright concerns. The "words"-file seems to be an index file (I have no idea about the encoding).
I had some plans about manipulating dictionaries (link) and I actually put some efforts into it. However, I gave up because of the encryption.
I still have a faint hope that the KT stores the html files in plain format on the system partition for performance reasons. Since I am a Ms Windows user without any knowledge of Linux I cannot access this partition. I would be thankful for any information.

[clsdclsd]

Originally Posted by tshering

I had some plans about manipulating dictionaries (link) and I actually put some efforts into it. However, I gave up because of the encryption.

I have done a similar thing what you planned: translated the 1.9.17's english monolingual dictionary's html files through a free web service to get english-hungarian translations, and rebuilt and recompressed the htmls. The "words" file was kept intact.
The good news are that even though the newer firmwares have encrypted dictionaries, my translated dictionary works perfectly with the 2.1.5 firmware, and also it is much more easy to install since the dictionary files are now on the user-reachable part of the memory.
So don't give up, finish your work :-)

[tshering]

Thank you so much for this information. Since my KT crashed completely when I tried to install my modified dictionary, I thought it was because of the encryption. But since it worked in your case there must be a different cause. Now I realize that the character encoding of the file names has changed since 2.1.1 or so. That gives me new hope. Thank you again.

[mnjkl]

I finally update my modified dictionary into firmware 2.15:

1 update kobo desktop to latest.
2 connect my kobo touch to my computer (my firmware is 1.9.17).
3 kobo desktop will prepare upgrade package for it.
4 under the .kobo folder of my reader, there is some file named dict???.gz.
5 open dict?.gz with a compressor (i am using 7z.exe)
6 replace files inside dict?.gz with my modified dictionary for 1.9.17.
7 disconnect ereader from my computer.
8 after all upgrade is finished, my modified dictionary is ready.

[tshering]

Originally Posted by mnjkl

I finally update my modified dictionary into firmware 2.15:

1 update kobo desktop to latest.
2 connect my kobo touch to my computer (my firmware is 1.9.17).
3 kobo desktop will prepare upgrade package for it.
4 under the .kobo folder of my reader, there is some file named dict???.gz.
5 open dict?.gz with a compressor (i am using 7z.exe)
6 replace files inside dict?.gz with my modified dictionary for 1.9.17.
7 disconnect ereader from my computer.
8 after all upgrade is finished, my modified dictionary is ready.

I am glad that your modified dictionary is working. I am still not able to do it with the Japanese dictionary. Could you tell me which dictionary you replaced? I am especially interested whether the recent (original) version consists of gzipped html files (signature 1F8B08), like the English-English dictionary, or of html files with another encoding (no signature), like the French-French dictionary.

[mnjkl]

English - french dictionary.

[tshering]

Thank you for the quick response. The html files of the English-French dictionary are (still?) gzipped. Did anybody successfully replace one of those dictionaries that have the other (to me unknown) file format?

[mnjkl]

it's same format as old firmware. it's a gz compressed html file.

[tshering]

Thank you again for the quick response. By "still?" I only wanted to indicate that I wonder whether they will change the format of all dictionaries in the near future. But maybe only some of the dictionaries are copyright protected, and this was the reason for encrypting them.

[AlPe]

Originally Posted by PeterT

Words looks to have a signature of "We love Marisa.".

Matching Algorithm with Recursively Implemented StorAge (MARISA) : http://code.google.com/p/marisa-trie/

[clsdclsd]

Originally Posted by tshering

Thank you again for the quick response. By "still?" I only wanted to indicate that I wonder whether they will change the format of all dictionaries in the near future. But maybe only some of the dictionaries are copyright protected, and this was the reason for encrypting them.

The last firmware version which had the dictionaries embedded was the 2.1.1, and all the dictionaries were encrypted in it. Now all of the installed ones on my Kobo are encrypted except the one I replaced: the english-english. Originally it was also encrypted, but as I used the dict from the 1.9.17 it is now simply gzipped. And works.

I assume you and mnjkl can have some non-encrypted dictionaries on the reader beacuse of skipping the 2.1.1, and the update procedure somehow kept the "old" dictionaries on the reader instead of replacing them with the new ones. All the dictionaries in the Desktop app are encrypted, so I am pretty sure a deselect (at language settings)-sync-reselect-sync procedure would result in all-encrypted dictionaries. In case the Kobo developers decide to not support the gzipped dictionaries in a future firmware I will downgrade to 2.1.5.

Anyway, I can't see why your japanese dict replacement isn't succsessful. You should try replacing the current (encrypted) japanese dict on your reader with the (gzipped) japanese from the 2.0.0 without editing it. If that works, that means your edited dictionary is somehow corrupted, but the replacement method works.
If it fails, that means the japanese dictionary is handled differently than the english-english. You can still try to replace another dictionary you are not using with the japanese-english and see what happens. It wouldn't be "elegant" to have to choose eg. dutch dictionary to translate from japanese but still...

[tshering]

Originally Posted by clsdclsd

The last firmware version which had the dictionaries embedded was the 2.1.1, and all the dictionaries were encrypted in it. Now all of the installed ones on my Kobo are encrypted except the one I replaced: the english-english. Originally it was also encrypted, but as I used the dict from the 1.9.17 it is now simply gzipped. And works.

This is really strange. As you say, in 2.1.1, the E-E dictionary is encrypted. In the recent dictionary files, as delivered in the synchronizing process, the E-E dictionary, and some others, the html-files are again gzipped. I thought this is the reason why yours is working fine.

Originally Posted by clsdclsd

You should try replacing the current (encrypted) japanese dict on your reader with the (gzipped) japanese from the 2.0.0 without editing it. If that works, that means your edited dictionary is somehow corrupted, but the replacement method works.

This is really a nice idea. I tested already with the encrypted dictionary whether I do the decompression and compression correctly. I was unsure of it because of the unicode file names. But it worked well. As for my edited dictionary, I doubt that I messed it up. And even if I did, the dictionary engine seems to be very forgiving, since already in the original files there are mismatching tags and they dictionaries work nonetheless. But I will certainly test it. Thanks for the good idea!

Originally Posted by clsdclsd

You can still try to replace another dictionary you are not using with the japanese-english and see what happens. It wouldn't be "elegant" to have to choose eg. dutch dictionary to translate from japanese but still...

I tried this already. I replaced the German-German dictionary (in the recent version, it is not encrypted). It did not work.

Would be interesting to replace the recent E-E dictionary with the version of 2.1.1 in order to see whether the dictionary engine can handle both gzipped and encrypted files in this case.

Thanks again for the input!

[clsdclsd]

Originally Posted by tshering

This is really strange. As you say, in 2.1.1, the E-E dictionary is encrypted. In the recent dictionary files, as delivered in the synchronizing process, the E-E dictionary, and some others, the html-files are again gzipped. I thought this is the reason why yours is working fine.

What about the files on your computer in the Kobo desktop app's directory? In my case those are exactly the same encrypted-zipped ones as the files on my reader - however the "old" non-zipped directories are also there. I assumed these are simply transferred during syncing, and that's why all my dictionaries were encrypted. (I synced via desktop not Wifi)

Originally Posted by tshering

Would be interesting to replace the recent E-E dictionary with the version of 2.1.1 in order to see whether the dictionary engine can handle both gzipped and encrypted files in this case.

Definitely can, as mine worked with the encrypted-english dictionary and now works with the gzipped-modified one.