New dictionary format of firmware 2.14

[tshering]

I am not sure whether I am inspecting the right files now. Yesterday, I factory reset my Touch and updated via desktop to 2.5.0. I selected all offered dictionaries and syncronized. Strangely, among the 19 dictionaries only two worked (Italian definition dictionary and Spanish definition dictionary; though I know that the English dictionary works for ShellShock). In the database all entries in the dictionary table looked ok to me. I manually installed 2.4.0 over 2.5.0 (note: without removing the new dictionaries), and everything is working again.
I just opened some of the dictionaries and see that the filestamp of the html files is rather old (07.08.2012 in dicthtml.zip and dicthtml-de.zip; the words files are even older than that). Is this the same with you or are my files somehow messed up?

Yesterday I already had a look at the new dictionaries. I was looking whether there are still some not encrypted. The Japanese dictionary was still plain text, but since it did not work on my Touch anyway, I thought there is not much to be learned from it about a new format or so (and I did not see anything new).
I am a little clueless right now.

German dictionary is still plain text (provided I am really looking at the new files; actually the html files are the same as in the "old" dictionary). Is this dictionary working under 2.5.0 on anyone's device?

[ShellShock]

Somehow I don't think you have the latest dictionaries. The html files in my 2.5 German dictionary (dicthtml-de.zip) are encrypted, but the Japanese (dicthtml-ja.zip) ones don't seem to be. Looking at the 2.1.4 versions, the German one is also are encrypted. Also the 2.5.0 German dictionary works fine for me on firmware 2.5.0. Its file size is 6.54 MB (6,866,837 bytes).

Looking at an html file in the Ja 2.5.0 firmware I can't see any formatting differences compared to my custom dictionary, apart from the Ja file is all one line. I have tried the same with my dictionary, and it still does not work.

[tshering]

Originally Posted by ShellShock

Somehow I don't think you have the latest dictionaries.

Thank you, ShellShock. That probably explains why the dictionaries did not work for me. Strange thing is that the dictionary-zip files in my desktop application folder have all the filestamp 29.03.2013, and this is in accordance with the information in the database. The dictionary files on my Touch, however, have all the timestamp 19.04.2013. But the content is much older than the content of the desktop dictionaries. One could think that some of the dictionaries were copied from the reset partition, and only those unavailable (in FW 1.9.16) were downloaded.
I just see that the Portuguese dictionary in the desktop application folder and the one on my Touch are in plain text. The file stamps of the htmls of this dictionary is 28.10.2012 (much younger than my reset FW 1.9.16). So this dictionary has evidently been downloaded yesterday after updating to 2.5.0. This might be the cause for this problem.
The new Italien dictionary too is in plain text.The file name convention seems a little different: 11.html has been split into 1a.html and so on. No more filenames with accented letters. The entries from ké.html, e.g., are gone to k1.html. Anyway, if this dictionary is working under 2.5.0, it might be a helpful case.

[mnjkl]

Maybe it is LZMA compressed file without header?

Or we can say it is a .7z file without header?

[ShellShock]

I tried all the compression formats supported by 7zip by renaming the .html extension of one of the existing Kobo dictionary files and seeing if they would open in 7zip. I did not have any success. But I did not remove any headers (I don't know how to). I suspect the files are gzip with some added encryption/obfuscation.

[tshering]

Could anybody please confirm whether the new Italian dictionary is working under 2.5.0?

[ShellShock]

Originally Posted by tshering

Could anybody please confirm whether the new Italian dictionary is working under 2.5.0?

Yes it is OK.

[AlPe]

Thanks for your observations. I will try to get the "new" Italian dictionary and inspect it.

[AlPe]

Originally Posted by davidfor

They might think so. But, as Kobo don't actually support custom dictionaries, it is more likely that the change was for some other reason.

I agree.

[ShellShock]

Good luck. I have looked at the new Italian dictionary, and it is "encrypted" but the Portuguese one dicthtml-pt is not. I have formatted my custom dictionary to match the Portuguese one, but it still does not work. The Kobo Portuguese dictionary also seems to be broken - looking up "ola" does not work.

[mnjkl]

I think kobo is using lzma sdk which can get a smaller size. But it is not using 7z header routine.

[ShellShock]

So, how do we write the file without the header? Does anyone know the size of the 7z LZMA header? I have been googling and can't get a clear answer. Is it even a fixed size? I have tried different sizes but no luck so far...

[tempest@de]

Originally Posted by ShellShock

The Kobo Portuguese dictionary also seems to be broken - looking up "ola" does not work.

it could be because it is «olá» and not «ola» maybe you could try again.

[ShellShock]

I don't think so. It could not find olaria either.

[tshering]

Originally Posted by ShellShock

I have looked at the new Italian dictionary, and it is "encrypted"

I am not sure whether this is helpful. Nevertheless, I post the beginning of the list of the content of the Italian dictionary that my Desktop application downloaded yesterday (Saturday night). Its size is 13.680.662. The html files are gzipped, the content is in plain text. The format is nothing special. But look at the file names!

Spoiler:

Saturday's Italian Dictionary
21.01.2013 16:34 940 1.html
21.01.2013 16:34 4.463 1a.html
21.01.2013 16:34 608 1b.html
21.01.2013 16:34 2.072 1c.html
21.01.2013 16:34 523 1d.html
21.01.2013 16:34 3.468 1e.html
21.01.2013 16:34 1.750 1f.html
21.01.2013 16:34 1.819 1g.html
21.01.2013 16:34 5.441 1i.html
21.01.2013 16:34 1.925 1l.html
21.01.2013 16:34 2.051 1m.html
21.01.2013 16:34 1.297 1n.html
21.01.2013 16:34 3.975 1o.html
21.01.2013 16:34 1.989 1p.html
21.01.2013 16:34 416 1q.html
21.01.2013 16:34 511 1r.html
21.01.2013 16:34 1.531 1s.html
21.01.2013 16:34 2.442 1t.html
21.01.2013 16:34 2.012 1u.html
21.01.2013 16:34 562 1v.html
21.01.2013 16:34 438 1z.html
21.01.2013 6:34 4.494 a.html
21.01.2013 16:34 563 a1.html
21.01.2013 16:34 44.600 ab.html
...

After reading that ShellShock's Italian dictionary is encrypted, I tried to download the Italian dictionary again via the desktop application. The application reported for a long time that it is downloading dictionaries, but actually no dictionary was downloaded; even after I repeated the process nothing changed. An inspection of the traffic showed however, that the application sent a GET /ereader/dictionaries/dicthtml-it.zip ... Host: download.kobobooks.com. I downloaded the file via the web browser. Its size is 11.206.504, it is encrypted. I guess this is the same as ShellShock's. Or maybe it is not? Note that the filestamps of the htmls of my Sunday Italian dictionary are older than those of my Saturday Italian dictionary.

Spoiler:

Sunday's Italian Dictionary
15.11.2012 19:45 26.512 11.html
15.11.2012 19:45 4.320 aa.html
15.11.2012 19:45 36.736 ab.html
15.11.2012 19:45 87.776 ac.html
15.11.2012 19:45 35.392 ad.html
15.11.2012 19:45 8.304 ae.html
15.11.2012 19:45 31.600 af.html
15.11.2012 19:45 36.368 ag.html
15.11.2012 19:45 656 ah.html
15.11.2012 19:45 4.944 ai.html
15.11.2012 19:45 90.112 al.html

Another question: Did somebody try to adjust the database entries of Size and LastModified in accordance with the values of a custom dictionary?