Custom Japanese-English dictionary

[tshering]

The original opening of this post is now out-of-date, I therefore put in into the spoiler. Start with the newest dictionary file (dicthtml-ja.zip). It should be ok for most purposes. If you really need to search the Japanese dictionary, install the two other files and read the spoilers. (In more recent firmwares, for instance 3.12.0, you can rename japdic01.zip and japdic02.zip to dicthtml-ja-en, dicthhtml-ja-de or similar; cf. also this post.)

Spoiler:

MOTIVATION FOR BUILDING A CUSTOM JAPANESE-ENGLISH DICTIONARY
The Japanese definition dictionary that is available for the Kobo Touch and newer models turned out to be of not much use to me. First reason is that my knowledge of Japanese is rather small and that I therefore need a translation dictionary. In an attempt to improve my situation, I added to the entries of the Japanese definition dictionary all matching entries from the EDICT. Only after this I became aware that the KT handles queries to the Japanese dictionary rather inadequately and that the chances to get the appropriate definitions are rather little. For an attempted explanation and a proposed solution, see this post. This solution requires a small change in the software and I hope we can see something on these lines in a future FW.
Even without changing the software, it is still possible to apply the basic idea of the proposed solution. There is, however, one server drawback without the change, and this is the high number of required chunks. Actually a decent Japanese dictionary requires more chunks than the format of the Kobo dictionary file format allows for. I therefore split the dictionary into two partial dictionaries.

CONTENT OF THE DICTIONARY
Part 1 (japdic01.zip) contains those entries of the EDICT that start with a Jouyou-Kanji.
Part 2 (japdic02.zip) contains
- all entries of the Edict that start with a Kanji beyond the Jouyou range
- all entries of the Edict that start with a Katakana character
- all entries of the whole Edict, except those starting with Katakana, but this time rearranged in a way that makes them available via Hiragana writing.
As you see, I did not especially treat those entries that have a mixture of Hiragana and Kanji AND start with a Hiragana.


INSTALLATION
In order to install this dictionary one has to sacrifice two dictionaries, other than the J-J and the F-F dictionaries (those two do not work properly). For the following explanation I presume that you sacrifice the Spain-Spain and the Dutch-Dutch dictionaries.

1) make sure that the Spain-Spain and the Dutch-Dutch dictionaries are properly installed on your reader.
2) delete the files dicthtml-es.zip and dicthtml-nl.zip in the directory {driveletter}:\.kobo\dict of your reader.
3) rename japdic01.zip to dicthtml-es.zip, and japdic02.zip to dicthtml-nl.zip. Copy both files into the directory {driveletter}:\.kobo\dict of your reader.

If the language of the epub/kepub is defined as Japanese and you point at a word, the original Kobo Japanese dictionary will pop-up, provided it is installed. In order to let japdic01 or Japdic02 pop-up set the book's language to Spain or Dutch (provided that book settings can be changed).
To make full use of the dictionary search function set the user language to Japanese, so that you can type in Japanese into the dictionary search field. (Edit: With FW 2.3.1. you cannot any longer type into the search window of the Netherlands dictionary. I therefore recommend now to rename japdic02.zip to something other than dicthtml-nl.zip.)

Consisting of 64.103 and 15.250 html(gz)-files respectively, japdic01.zip and japdic02.zip are far more fragmented than the other dictionaries. Since I tested them only for a very short time I cannot say how they influence the overall performance or the power consumption of the reader.

NEW: With FW 2.3.1 the dictionary engine for Japanese has improved a lot. I therefore provide the EDICT for this version. In addition to the materials available in japdic01.zip and japdic02.zip it contains also all entries starting with fullwidth Latin characters (e.g., "S造" and "100円玉").
Unfortunately one has to replace the original dicthtml-ja.zip by this one, since renaming the file prevents it from functioning. A possibility to use the information of both dictionaries (the one provided by Kobo and the Edict) is to merge them into one. For copyright concerns I do not provide such a merged dictionary here.
As it is even under FW 2.3.1 impossible to type into the search window of the Japanese dictionary, japdic01.zip and japdic02.zip are still more convenient to use in many cases.

Spoiler:

See this post for some tips on how to use the three files together to get the most out of them.

DISCLAIMER
You assume total responsibility and risk for your use of the provided files; use at your own risk.

ACKNOWLEDGEMENT
This package uses the EDICT dictionary file. This file is the property of the Electronic Dictionary Research and Development Group, and is used in conformance with the Group's licence.

I would like to thank several forum members and especially those posting at this thread for advice and encouragement.

EDIT: With more recent firmwares, the name of the Japanese dictionary has been changed to dicthtml-jaxxdjs.zip (instead of dicthtml-ja.zip).

[andrusz]

tshering I was searching for jap-eng dictionary for Kobo but there is none on the market. I have even tried creating one based on EDICT but have some issues. Now I am going to test yours. Will let you know about the results.

ありがとう！

[andrusz]

According to your suggestions I have installed japdict01 as Portugues dictionary (which I had already installed) and it works very well :-) The only drawback is my Kobo Touch is using original Japanese dictionary every time I am searching for a kanji while reading a book. So I have to click "Dictionary" icon and change dictionary from Japanese to Portuguese. I have to do it for every kanji, it is little frustraiting...

So I have tried to replace Japanese dictionary with japdict01 (or japdict02) but it does not work for me. Kobo cannot find any kanji :-( Could you explain how to do it, if it is possible?

Finally I have added japdict01 as a new dictionary jap-eng. It works very well, too :-) However the dictionary availability depends on the book. Some books allows you to choose "Translation dictionary" but most I have (from Aozora) does not. Very odd.

tshering, thanks for a good job!

[tshering]

Originally Posted by andrusz

According to your suggestions I have installed japdict01 as Portugues dictionary (which I had already installed) and it works very well :-) The only drawback is my Kobo Touch is using original Japanese dictionary every time I am searching for a kanji while reading a book. So I have to click "Dictionary" icon and change dictionary from Japanese to Portuguese. I have to do it for every kanji, it is little frustraiting...

The reader chooses the dictionary in accordance with the language settings of the book. This is why the Kobo Japanese dictionary pops up in case of Japanese books. In order to prevent this you can change the books settings. In your case you should set the language to Portuguese. With DRM free epubs this is rather easily done. In other cases you have to find your ways around. You might use Calibre for changing the settings and there is also a plug-in for the DRM problem. You can find more information on this forum. I cannot say more about it, since I do not use Calibre myself.

Originally Posted by andrusz

So I have tried to replace Japanese dictionary with japdict01 (or japdict02) but it does not work for me. Kobo cannot find any kanji :-( Could you explain how to do it, if it is possible?

The original Japanese dictionary works internally different from the other dictionaries (and this is actually its weak point). Therefore, you cannot use it for our purpose.

Originally Posted by andrusz

Finally I have added japdict01 as a new dictionary jap-eng.

How did you do this?

Originally Posted by andrusz

It works very well, too :-) However the dictionary availability depends on the book. Some books allows you to choose "Translation dictionary" but most I have (from Aozora) does not. Very odd.

I would recommend using japdic01 and japdic02 both as definition dictionaries rather than one as translation dictionary (I personally opted for Spain and Dutch). If the language of a book is defined as Japanese (and you cannot or do not want to change it) the reader (usually?) does not offer you any translation dictionaries. The reason is that there is no original Japanese translation dictionary. Another advantage of having both defined as translation dictionaries is that in some situations it is easier to change from one translation dictionary to another translation dictionary as it is to change from a definition dictionary to a translation dictionary.

[andrusz]

Originally Posted by tshering

The reader chooses the dictionary in accordance with the language settings of the book. This is why the Kobo Japanese dictionary pops up in case of Japanese books. In order to prevent this you can change the books settings. In your case you should set the language to Portuguese.

Good idea! I have just changed the language of my book and the dictionary works fine

Originally Posted by tshering

The original Japanese dictionary works internally different from the other dictionaries (and this is actually its weak point).

What I have noticed:
- old format Japanese dictionary contains UTF-8 encoded filenames in the dicthtml-ja.zip file. But the files (descriptions) are encrypted:


- new format Japanese dictionary contains Shift_JIS encoded (this is my guess - I cannot verify it) filenames in the dicthtml-ja.zip file. The files, however, are standard UTF-8 HTML definition files (gzipped):


So it seems that changing the filenames in your dictionary from UTF-8 to Shift_JIS should "convert" translation dictionary to Japanese dictionary The "words" index file is UTF-8 uncoded in both dictionary versions (the file is exactly the same, in fact) so it does not need to be changed.

Finally I have added japdict01 as a new dictionary jap-eng.

You just have to put a dictionary file do dict folder with a proper name, e.g. dicthtml-ja-en.zip. That's all It will not be listed in the Settings screen but will be shown as "Translation dictionary" while reading a book.

If you want the dictionary to be also shown in the Settings screen you need to add a line to "Dictionary" table in KoboReader database (values similar to other installed dictionaries, except "Suffix" column and "Name", of course).

Originally Posted by tshering

If the language of a book is defined as Japanese (and you cannot or do not want to change it) the reader (usually?) does not offer you any translation dictionaries.

This is a limitation of the Kobo firmware, IMHO. It will not offer a translation dictionary even if you remove Japanese dictionary

Originally Posted by tshering

I would recommend using japdic01 and japdic02 both as definition dictionaries rather than one as translation dictionary (I personally opted for Spain and Dutch).
[...]
Another advantage of having both defined as translation dictionaries is that in some situations it is easier to change from one translation dictionary to another translation dictionary as it is to change from a definition dictionary to a translation dictionary.

You are right!

[tshering]

Originally Posted by andrusz

I have just changed the language of my book and the dictionary works fine

I am glad to hear it!

As for the filenames. According to my understanding, under FW 2.0.0 filenames were UTF-8 encoded, with later FWs they were 2-byte unicode encoded. And this is how I encoded them in my dictionary (namely 2-byte unicode). But sometimes it is hard to say what kind of encoding conversions are executed behind the screen (by the OS or decompressing tools.)

By saying

Originally Posted by tshering

The original Japanese dictionary works internally different from the other dictionaries (and this is actually its weak point). Therefore, you cannot use it for our purpose.

I meant to say that if a dictionary file is named dicthtml-jp.zip the reader will call a search routine that works very special. It matches Kanjis with their Kana representation and then searches for the Kana reading. For instance, if you search for 本 the search engine will try to find it in the file ほん.html or maybe もと.html. In japdic01.zip, however, the entry is located in 本a.html. This is the reason why japdic01.zip cannot correctly work if you rename it to dicthtml-jp.zip.

Originally Posted by andrusz

You just have to put a dictionary file do dict folder with a proper name, e.g. dicthtml-ja-en.zip. That's all It will not be listed in the Settings screen but will be shown as "Translation dictionary" while reading a book.

Does not work for me. Which FW are you on?

[andrusz]

Originally Posted by tshering

As for the filenames. According to my understanding, under FW 2.0.0 filenames were UTF-8 encoded, with later FWs they were 2-byte unicode encoded. And this is how I encoded them in my dictionary (namely 2-byte unicode). But sometimes it is hard to say what kind of encoding conversions are executed behind the screen (by the OS or decompressing tools.)

I think new Japanese dictionary has Shift_JIS filenames (used by MS Windows Japanese version). I am going to install Japanese Windows and investigate it more deeply...

Originally Posted by tshering

I meant to say that if a dictionary file is named dicthtml-jp.zip the reader will call a search routine that works very special.

I understand. So it requires more work to prepare one

Originally Posted by tshering

Does not work for me. Which FW are you on?

My Kobo has firmware 2.2.0. Be sure the name contains known language abbrevations like dicthtml-ja-nl.zip or dicthtml-ja-es.zip

You can easily check it is installed:
- open any english book
- activate a dictionary by touching a word
- click on A/Z icon and choose "Translation dictionary"
Your new dictionary should be on the list.

[tshering]

Originally Posted by tshering

As for the filenames. According to my understanding, ..., with later FWs they were 2-byte unicode encoded.

I have to confess that this is nonsense. The filenames are stored in the zip file in UTF-8 encoding. You can verify this by opening the zip file in a HEX editor and inspecting the filenames. When the files are unzipped the encoding of the filenames depends on the settings of the unzipping program and the OS. 7-zip under windows encodes them by default as UTC/unicode.

[andrusz]

Originally Posted by tshering

I have to confess that this is nonsense. The filenames are stored in the zip file in UTF-8 encoding. You can verify this by opening the zip file in a HEX editor and inspecting the filenames. When the files are unzipped the encoding of the filenames depends on the settings of the unzipping program and the OS. 7-zip under windows encodes them by default as UTC/unicode.

I do not have hex editor right now to verify it but look at my previous post. This is a list of zip file contents run on RedHat Linux with UTF-8 language settings using the following command: unzip -l dicthtml-ja.zip

I have observed the same on Windows 7 (english version) opening the files using Winzip:

Old dictionary:


New dictionary:


Have you observed the same?

[tshering]

andrusz, thank you for the images. Now I see that we talked about different things. When I talked about the old version I meant the first one that was contained in KoboRoot.tar of kobo3-update-2.0.0.zip. And what I called new version is your old version. I got different results with different tools.
I give below as an example the results for one and the same file extracted with differnet tools from 1) kobo3-update-2.0.0.zip, 2) your old version, 3) your new version.

1) kobo3-update-2.0.0.zip
7zFM (7-zip GUI): ÒüôÒüå.html 12.07.2012
GnuWin32\bin\tar.exe: ��.html 12.07.2012 (this is utf8)

2) dicthtml-ja (old version)
7zFM (7-zip GUI): こう.html 07.08.2012 Host OS: Unix
Windows zip (as used by the file explorer): ÒüôÒüå.html 07.08.2012

3) dicthtml-ja (new version)
7zFM (7-zip GUI): ÒüôÒüå.html 12.11.2012 Host OS: FAT
Windows zip (as used by the file explorer): ÒüôÒüå.html 12.11.2012

I do not know what kind of encoding ÒüôÒüå is. Inside the html.gz-file the filename is again encoded in utf-8.

It is nice that the dictionary zip file contains now the source file for "words" too!

[andrusz]

Originally Posted by tshering

I do not know what kind of encoding ÒüôÒüå is. Inside the html.gz-file the filename is again encoded in utf-8.

I have finally resolved this by using MAC OS X It compreses (zip) the files to format which can be read by Kobo. I have created a test jap-eng dictionary and it works fine as Japanese dictionary

Originally Posted by tshering

It is nice that the dictionary zip file contains now the source file for "words" too!

Yes, it is nice but I cannot create a proper words file for my dictionary Could you help me creating a good one?

My source list looks like the words.original file. I am using Marisa to create words file and it works ok using marisa-lookup or marisa-predictive-search tool. But when the file is added to my test dictionary (to zip file) it does not work in Kobo Using original words file with my dictionary works ok, however. It means my words file is wrong.

What I have observed using Marisa tools is a different output using original Kobo words file comparing to mine. Below is the output from marisa-predictive-search.

Original file:

いえ
18 found
71      いえ    いえ
3502    いえ【家】      いえ
10968   いえい【遺影】  いえ
24088   いえがまえ【家構え】    いえ
24089   いえがら【家柄】        いえ
10969   いえき【胃液】  いえ
10970   いえじ【家路】  いえ
10971   いえで【家出】  いえ
10972   いえども【雖も】        いえ
10973   いえなみ【家並み】      いえ

My file:

いえ
43 found
58634   いえい【遺影】
58635   いえい【遺詠】
37152   いえい【家居】
37153   いえいえ【否否・否々】
37154   いえいえ【家々・家家】
37155   いえつき【家付き】
37156   いえつきのむすめ【家付きの娘】
37157   いえつきむすめ【家付き娘・家付娘】
18160   いえつづき【家続き】
58636   いえにかえる【家に帰る】

[tshering]

In the marisa output of your file, there is "いえ" missing at the end of each line. I guess therefore that there is something at the end of each line that prevents showing the rest of the line. Did your check that in your file all lines end on LF (without CR)?

[andrusz]

Originally Posted by tshering

Did your check that in your file all lines end on LF (without CR)?

You are right! My word list was produced on Windows and it was the issue. The list created on Mac works fine

[andrusz]

Below is my Japanese-English dictionary based on EDict2. It has been created as a replacement of original Kobo Japanese dictionary.

To install just copy (and replace) dicthtml-ja.zip file in .kobo/dict folder in your Kobo reader.

It would be a good idea to do a backup of the dicthtml-ja.zip file before overwriting.

Japanese-English dictionary

Warning: The file cannot be used as Japanase-English translation dictionary. It means you cannot change the name of the file and add it as an additional Kobo dictionary - it will not work this way!

[tshering]

Great work, Andrusz! However, I would like to repeat that the search mechanism used for dicthtml-ja does not do a good job. For an explanation, see this post. You can verify what I am saying by pointing at, e.g., 調. With your dictionary loaded, the reader will provide as the closest match "しらべ [白檜] (n) (uk) Veitch's silver fir (Abies veitchii)/". The correct answer, however, is: "ちょう [調] /(n) (1) pitch/tone/key/(2) time/tempo/(n,suf) (3) mood/tendency/style/(n) (4) (arch) tax on products/". Note: My KT is still on FW 2.1.5. If the handling of the Japanese dictionary has improved under a more recent FW, please tell me. Because then I would manually upgrade.