Index of Custom Dictionaries for Kobo eReader

[Simpetus]

Ok, now it is really strange.

At the start I have made short test version of Chinese to Russian dictionary in Kobo format that consist of only two characters - 谁 and 当时. And named it "dicthtml-es-en.zip". I turned on my Kobo H2O opened a book in Chinese (Chai Jing - Kan Jian - Chai Jing.kepub.epub), I successfully found and checked those two characters in the text (谁 and 当时) - worked flawlessly and their translations displayed in pop-up dictionary window.

After that, I converted full version in .dict format (zho_rus.dict) into Kobo's in the exact same way as with short test version and gave it the same name, put in the .kobo dictionary folder and replaced short version with full one.

Then I turned on my eBook to find out that Kobo cannot see the full version of the dictionary and displays that I have not es-en dictionary installed. What the hell? I have tried to rename full version to "dicthtml-de-en.zip","dicthtml-fr-en.zip"...you name it and still nothing, with every new name of Kobo's system just keep say that "I do not have (Language) of the dictionary installed in my system.

I started to think that "I probably done something wrong during the process" and decided to download custom Chinese-English dictionary (https://www.mobileread.com/forums/sho...d.php?t=202182) to check whether it is true and I really messed up again.

After I downloaded the file I replaced default dictionary with this one and guess what?.. Kobo still keeps saying that "I do not have (name)dictionary installed in my system".

So, where did I made a mistake? And why short version works, but full one not?

I am desperate.

I am providing all files that need to make a test:

- original book in chinese in kepub (it is free, no worries);
- short and full version of the Chinese-Russian Kobo format dictionaries;
- original stardict Chinese-Russian dictionary (in case I have made mistake during converting stardict to kobo's format).

Please, help, I love nickel more than KoReader.

Thanks in advance

P.S. If you need me to upload the archive to another sharing service, just say it, I will do it right away.

https://mega.nz/#!fwczhDLZ!qZTkv5_8e...csYkmfakV6d1DE

[Simpetus]

Originally Posted by AlPe

EDIT: I missed the message above. Good. Anyway, updating Penelope will not hurt. v1.20 is a really old version.

BTW, I converted all versions of dictionaries with new Penelope version 2.0.2.

[Simpetus]

I even downloaded SQLite Editor and tried to play with parameters in KoboReader.sqlite file:

1. Checked values - Installed and Synced are all "true";
2. I have corrected the size (no mistake in bytes I was very careful and precise) of the custom dictionary saved all changes and start the dictionary - no joy, Kobo's system keeps say that I have no the dictionary installed;
3. I even add new entry in Dictionary section of the SQ-file with suffix "zh-ru", exact byte size, set all parameters on "true" and change the name to "dicthtml-zh-ru.zip" - it does appeared in dictionary section when I started Kobo, but when I tried to search a word it was not in the dictionary pop-up window list.

Damn you Kobo

[Simpetus]

Damn I want to use Nickel and read Chinese books with Chinese dictionary present.

[AlPe]

Hi,

I understand the frustration. The problem is the lack of official support for custom dictionaries. I tried to contact Kobo developers several times over the last 2+ years, without any result. (Same story for Bookeen, BTW.) When people send me emails asking for help with Kobo dictionaries, I help them if I can, but I also tell them to send a complaint email to Kobo customer service, asking for the feature to be implemented or at least publishing official documentation.

Back to your issue: I cannot exclude it is a problem with your full dictionary (either the input file, or the output of Penelope), albeit I doubt this is the problem. It seems something went wrong when you copied the full dictionary over the test (2 words) dictionary. Perhaps the words index file has not been replaced properly or at all.

You might want to try the following two procedures:

PROC 1

1. Remove/deselect all dictionaries in the Settings, and delete them from the .kobo/dict directory
2. Reboot your Kobo
3. Copy again the TEST (two words) dictionary
4. Open your eBook, and check if the word lookup works, as it did the first time you tried it

If not, then something really bad happened... reset your Kobo maybe?

If it worked, do the following:

PROC 2

1. Remove all dictionaries, as above
2. Reboot, as above
3. Copy the FULL dictionary
4. Open your eBook, and check if the word lookup works, using the SAME word that you used in the test with the 2 word dictionary

If this does work, good. If it does not work, then it might be an issue with your full dictionary file.

[Simpetus]

Originally Posted by AlPe

Hi,

I understand the frustration. The problem is the lack of official support for custom dictionaries. I tried to contact Kobo developers several times over the last 2+ years, without any result. (Same story for Bookeen, BTW.) When people send me emails asking for help with Kobo dictionaries, I help them if I can, but I also tell them to send a complaint email to Kobo customer service, asking for the feature to be implemented or at least publishing official documentation.

Back to your issue: I cannot exclude it is a problem with your full dictionary (either the input file, or the output of Penelope), albeit I doubt this is the problem. It seems something went wrong when you copied the full dictionary over the test (2 words) dictionary. Perhaps the words index file has not been replaced properly or at all.

You might want to try the following two procedures:

PROC 1

1. Remove/deselect all dictionaries in the Settings, and delete them from the .kobo/dict directory
2. Reboot your Kobo
3. Copy again the TEST (two words) dictionary
4. Open your eBook, and check if the word lookup works, as it did the first time you tried it

If not, then something really bad happened... reset your Kobo maybe?

If it worked, do the following:

PROC 2

1. Remove all dictionaries, as above
2. Reboot, as above
3. Copy the FULL dictionary
4. Open your eBook, and check if the word lookup works, using the SAME word that you used in the test with the 2 word dictionary

If this does work, good. If it does not work, then it might be an issue with your full dictionary file.

1. About complain. I have send them three already:

1.a. About adding support for Stardict;

2.b. About make convert into Kobo's format app for Stardict format dictionaries;

3.c. About releasing documentation on dictionaries.

I pissed off because of all this situation really. I bought this eBook, they got my money and still I am not the owner and not allowed to use all functions of the eBook as suitable for me.

Yeah, I know that they probably aware of potential law suits be cause of users can use copyright\stolen dictionaries, but hell, they totally can write down statement in user agreement part, mentioning that all responsibility is on end user.


2. About whether there is a problem with final dictionary. As I said, test dictionary (that consist only of 2 characters) works just fine, and the final BIG version was converted in the same way as test version, so no issues with them or way I converted the dictionaries.

Anyways, thanks to another user (tshering) I found out the reason Kobo's system keep saying that there is no dictionary installed - the amount of html files in zip archive of the final version of the custom Chinese dictionary is too high - almost 120 000 html files.

So, even if you mentioned that, it seems there are no restrictions in file size, there are some about amount of html files in archive.

Later I extract all files in archive, deleted almost all of them, packed it again and put in the dictionary - it worked.

Any workarounds? tshering said to put all dictionary articles with similar characters into one html file, combine them, but:

1. I do not know how to automate the process;

2. And If I do it manually, with 120 000 html files in the archive it will take ungodly amount of time.

Originally Posted by tshering

@Simpetus
Edit: Actually, the short version is working! It was only my mistake.

Edit 2: I did a further test. I removed from your complete dictionary most html files, so that only 100 or so remained. This shortened version is working. I think this shows that your dictionary is properly made, but that there are more html files than the reader can handle. If you look at my first Japanese dictionary, I had to split it into 2 parts for this reason. But this of course was very inconvenient.
I guess, it is best to redo your dictionary according to the scheme for Japanese dictionaries, in order to get a smaller number of html files. Instead of, for instance 上心.html, 上情.html and so on, you would put all entries starting with 上 into one 上.html. Look at cedict4kobo.zip for the details.
I do not know whether penelope has a setting for this, or whether you have to achieve this somehow by yourself.

[davidfor]

Originally Posted by Simpetus

1. About complain. I have send them three already:

1.a. About adding support for Stardict;

2.b. About make convert into Kobo's format app for Stardict format dictionaries;

3.c. About releasing documentation on dictionaries.

I pissed off because of all this situation really. I bought this eBook, they got my money and still I am not the owner and not allowed to use all functions of the eBook as suitable for me.

Yeah, I know that they probably aware of potential law suits be cause of users can use copyright\stolen dictionaries, but hell, they totally can write down statement in user agreement part, mentioning that all responsibility is on end user.

Sorry, but you have no justification for this. At no point does Kobo claim the devices support random languages or dictionaries. The fact that some users have reverse engineered the device and worked out how some of this works, does not imply any promise of support from Kobo. You can try it, but if it doesn't work, it is your problem, not theirs.

Of course, I wish Kobo did support it. Or some other method of getting different dictionaries. When they changed the dictionary support, I thought Kobo would follow this by selling better or extra dictionaries.

[Simpetus]

Kobo has Chat system. http://kobo.frontlinesvc.com/app/chat/chat_launch/

[PeterT]

A chat with Kobo support will not serve any purpose here at all.

[oren64]

I Try to edit the dictionary in text file and convert it with penelope, but it didn't work it not stop creating HTML files +100k.
search for another dictionary Chinese to Russian.


I find Russian to Chinese dictionary and convert it with no problems.

абиссаль	名词 深海
абиссальный	形容词 深水的
абиссинец	〔阳〕阿比西尼亚人(埃塞俄比亚人的旧称). ‖абисс`инка, 复二-нок〔阴〕.
абиссинский	〔形〕阿比西尼亚(人)的.
абитуриент	<旧>中学应届毕业生\n〔阳〕⑴〈旧〉中学应届毕业生. ⑵升大学的人. ‖абитури`ентка, 复二-ток〔阴〕.

[tshering]

Originally Posted by oren64

I Try to edit the dictionary in text file and convert it with penelope, but it didn't work it not stop creating HTML files +100k.
search for another dictionary Chinese to Russian.


I find Russian to Chinese dictionary and convert it with no problems.

абиссаль	名词 深海
абиссальный	形容词 深水的
абиссинец	〔阳〕阿比西尼亚人(埃塞俄比亚人的旧称). ‖абисс`инка, 复二-нок〔阴〕.
абиссинский	〔形〕阿比西尼亚(人)的.
абитуриент	<旧>中学应届毕业生\n〔阳〕⑴〈旧〉中学应届毕业生. ⑵升大学的人. ‖абитури`ентка, 复二-ток〔阴〕.

The point is not whether the dictionary contains any Chinese signs, but whether the source language is Chinese (Chinese to Chinese, Chinese to English, and so on.) The problem is that Chinese (as well as Japanese and Korean) has much more characters than, for instance, English. Take a set of 20 to 30 characters (for instance, a-z) and build all possible pairs (aa, ab, ac, ..., of course not all pairs will be actually occur in the onset of a given language), and do than the same with a set of thousands of Characters (Chinese). As you see, the filename convention of [first character of the expression][second character of the expression].html leads to an tremendous amount of files in Chinese/Japanese/Korean dictionaries.

[AlPe]

EDIT: I wrote at the same time as Tshering, essentially saying the same.

Originally Posted by Simpetus

Anyways, thanks to another user (tshering) I found out the reason Kobo's system keep saying that there is no dictionary installed - the amount of html files in zip archive of the final version of the custom Chinese dictionary is too high - almost 120 000 html files.

That's because the reverse engineering effort has started from the Kobo dictionaries on latin scripts. In them each .html file inside the ZIP container (which is an HTML file, GZIPed and renamed removing the .gz extension) contains all the words stating with the same 2 characters prefix. Plus, there is some special rule for 1 character words. See:

https://github.com/pettarin/penelope...t_kobo.py#L132

Now, when I made Penelope I assumed that that was the Kobo convention. Apparently, as Tshering's dictionary shows, the Kobo firmware seems to deal with an arbitrary partitioning of the index into prefixes (.html files).

Of course grouping together by the first char leads to less .html files than grouping together by the first two chars. And this is especially true for languages like Chinese or Japanese where there are thousands of "characters".

When you lookup a word using a Kobo dictionary, there are three steps:

1. lookup in the MARISA trie ("words"), which essentially decides whether the word is present in the dictionary; assuming the word is present:
2. locating the .html file inside the ZIP container which contains the definition;
3. gunzipping the .html file, parsing the uncompressed HTML contents, locate the actual definition to show to the user.

The splitting policies essentially decides a tradeoff between 2. and 3. If you group by 1 char prefix, you will have a faster 2., but a slower 3. than grouping by 2 char prefix. I guess the 2 char prefix rule followed by Kobo for languages with latin scripts is the result of some experimentation, and they found it to be optimal for latin scripts. Since grouping by k prefix leads to order n^k .html files (where n is the number of base character), it works for latin script languages, as n~30, k~2 => ~900 pairs (usually much less, since not all possible prefixes occur in the language). As mentioned above, k=2 leads to explosion for languages like Chinese or Japanese.

Originally Posted by Simpetus

So, even if you mentioned that, it seems there are no restrictions in file size, there are some about amount of html files in archive.

It might be related to the fact that ZIP archive can have a maximum number of entries and/or it might be due to a limitation in the ZIP library that Kobo is using. Again, without documentation is not really possible to know, other than trying to build increasingly large dictionaries (with an increasing number of .html files), and find out the number of .html entries that makes the dictionary not working.

Originally Posted by Simpetus

Later I extract all files in archive, deleted almost all of them, packed it again and put in the dictionary - it worked.

Any workarounds? tshering said to put all dictionary articles with similar characters into one html file, combine them, but:

1. I do not know how to automate the process;

2. And If I do it manually, with 120 000 html files in the archive it will take ungodly amount of time.

Currently Penelope uses the "two chars" rule described above. But I can easily add a command line option to let the user specify the length of the prefix for grouping. I'll do it during the weekend.

[AlPe]

Originally Posted by davidfor

Sorry, but you have no justification for this. At no point does Kobo claim the devices support random languages or dictionaries. The fact that some users have reverse engineered the device and worked out how some of this works, does not imply any promise of support from Kobo. You can try it, but if it doesn't work, it is your problem, not theirs.

Surely true from a legal point of view, but...

Originally Posted by davidfor

Of course, I wish Kobo did support it. Or some other method of getting different dictionaries. When they changed the dictionary support, I thought Kobo would follow this by selling better or extra dictionaries.

... business-wise, this is the relevant point, and I personally think Kobo has not got it right.

It does cost (near to) nothing for them to at least publish a format specification, especially since it is well know (even to them!) that their format has been reversed engineered, and they could have bragged about it at length if they implemented it. The ROI is quite large, hence they should have done it. Clearly, either they think the ROI is not so large, or they have other priorities with higher ROIs.

BTW, the same is true for Bookeen, whose Odyssey dictionary format has been known and supported by Penelope for a while now. Bookeen even suggests to use Penelope to their users on Facebook and Twitter, when they ask for dictionaries in languages different than the official ones. Despite the fact that I have asked them to put somewhere, officially or privately, a format specification!

(Of course the fact that Kobo hard-coded the dicthtml*zip file names in their firmware is not really a good practice and, going "official", they might force them to solve that issue, which is a cost for them; Bookeen got that point right, as the Odyssey devices support arbitrarily named files for arbitrary languages.)

I suggest complaining with their customer care not because I hope that doing so will help (I don't think it will), but to "educate" the users to think about their choices. Of course we --- those discussing this issue here on a dedicated forum --- have way more technical knowledge than the average user. But, getting emails from "non-tech-savvy people" I can assure you that, for some of them, my explanation of why their custom dictionary does not work and my invitation to complain with Kobo customer care are quite a revelation....

[Simpetus]

Originally Posted by AlPe

Currently Penelope uses the "two chars" rule described above. But I can easily add a command line option to let the user specify the length of the prefix for grouping. I'll do it during the weekend.

Big thanks Albert for such quick and detailed answer!

Ok, I think I got it, the question is what length should we chose then?

[AlPe]

Well, assuming the problem with your dictionary is indeed the number of .html files too large, you will try with k=1 (1 char prefix) instead of k=2 (current rule), as it yields the minimum number of .html possible, given the set of words in your input dictionary.

If the resulting (k=1) dictionary works, you are done. If not, then your dictionary is too big anyway.

(There is another possibility to reduce the number of .html files further: merging together groups. E.g. (using latin for simplicity): instead of having a.html, b.html, c.html, ... , z.html , in theory one can build a.html (containing a*, b*, c* words), d.html (d*, e*, f*, g*), h.html (h*, ...), etc.). But I will be surprised this will work on Kobo.)

EDIT: can you send me just the "words" file of your full dictionary by email (or send me a link to it by email)? I will use it to test that the new option produces working dictionaries. Just the "words" file, I do not need the definitions. My email is here: http://www.albertopettarin.it/contact.html