New dictionary format of firmware 2.14 - Page 5

tshering · 11-13-2012, 05:44 PM

Thank you, ShellShock!

ShellShock · 11-14-2012, 05:08 PM

Just a quick update - I have done a first pass conversion of the Chambers Twentieth century dictionary circa 1913 on Project Gutenberg so it now runs on my Glo. It is still very buggy: can't find 1 or 2 character words, links in definitions to other words do not work, lookup for e.g., "hand" complains about not finding it and suggests the closest match as "Hand", still claims the Source is "Merriam-Wedtser" etc etc.

On the last point - I had a poke around the Sqlite db on the reader; there is a dictionary table on there, but it doesn't seem to hold all the information, e.g., no "Source". I tried adding a new record for my custom dictionary, but the reader did not recognise it, so I am still replacing the existing dicthtml.zip with my custom dictionary.

tshering · 11-15-2012, 07:05 AM

Quote:

Originally Posted by ShellShock

can't find 1 or 2 character words

If you put them into the appropriate html(gz) file and also into "words" then this behavior is certainly strange.

Quote:

Originally Posted by ShellShock

links in definitions to other words do not work

Could you please point me to links in definitions to other words that work in any Kobo-dictionary?

Quote:

Originally Posted by ShellShock

lookup for e.g., "hand" complains about not finding it and suggests the closest match as "Hand"

If the first letter is written with a capital letter in the "words" file (as it is written in Chambers's Twentieth Century Dictionary) this behavior seems understandable.

Quote:

Originally Posted by ShellShock

still claims the Source is "Merriam-Wedtser"

It seems that the device associates the copyright information, the language, the way it handles the queries (for instance, removing potential plural suffixes and so on if the search string is not found as it is) with certain filenames. In earlier versions of FW, there was no dictionary table in the db.

ShellShock · 11-15-2012, 07:33 AM

Quote:

Originally Posted by tshering

If you put them into the appropriate html(gz) file and also into "words" then this behavior is certainly strange.

It was a bit of a rushed job so I probably made some silly mistake. Will check tonight.

Quote:

Could you please point me to links in definitions to other words that work in any Kobo-dictionary?

Yes, I've noticed that too now. So something I do not have to worry about (getting links working, because they are not supported).

Quote:

If the first letter is written with a capital letter in the "words" file (as it is written in Chambers's Twentieth Century Dictionary) this behavior seems understandable.

Thanks, I'll check the capitalisation.

ShellShock · 11-15-2012, 05:05 PM

I have solved the mystery of why word lookup in my custom dictionary was not working - I could not look up any words shorter than 4 characters. Also marisa-lookup always returned -1 no matter what.

I had to get up close and personal with Marisa, the naughty thing, and put her through a debugger. This showed that all my keys had a carriage-return on the end. Ah ha - Marisa only plays nice with Unix, the hussy! After stripping all my carriage-returns (leaving just newlines) from my index.txt file, Marisa was much happier, and my new dictionary is now fully working.

I have also discovered that the html input files can have multiple <p> elements per <w> element, if you have multiple definitions for the same word, e.g.,

Code:

<w>
<p><a name="a"/><b>a</b>, the indefinite article</p>
<p><a name="a"/><b>a</b>, ä, a dialectic corruption of <i>he</i> or <i>she</i></p>
</w>

On the Kobo, the multiple definitions will be displayed on the same page, with a blank line between them.

The Chambers dictionary I used as my source only has 33,000 definitions (but I like them!), so I am going to look for something a bit more substantial to convert to the Kobo.

Terisa de morgan · 11-15-2012, 05:20 PM

Only a "small" comment: Marisa is a real Spanish nichname

AlPe · 11-17-2012, 10:53 AM

Quote:

Originally Posted by Terisa de morgan

Only a "small" comment: Marisa is a real Spanish nichname

Marisa is a female name in Italian as well. Usually having a grumpy, gossipy character.

===

To return in topic: inside the <w> tag, besides at least one <a name="key"> anchor, you can put pretty much whatever* HTML you want.

* Although I haven't tested tables.

shutramp · 11-17-2012, 03:16 PM

Would be possible to make one for chinese as well?

AlPe · 11-17-2012, 05:27 PM

Well, I guess that if it works for Japanese should work for Chinese as well --- but I am not a Unicode Master, so do not take my word for granted.

tshering · 11-17-2012, 05:59 PM

Quote:

Originally Posted by shutramp

Would be possible to make one for chinese as well?

Generally speaking, it is possible. But I am not sure whether it is practicable. The dictionary entries are distributed among several html files according to the following rules:
1) If a word/expression consists of one letter "X" it goes into the html file with the name "Xa.html".
2) If a word/expression consist of more than one letter "XYZ" the name of the html file is made from the first 2 letters "XY.html".
3) If the first or second char is not a "letter", but a hyphen, a number and so on it goes to "11.html".

Example for case 1: "漢" goes into "漢a.html"
Example for case 2: "漢字" goes into "漢字.html"
I tried this with a very small sample of a Japanese dictionary (disguised as German dictionary), and it worked.

The problem that I see is the huge number of html files that are needed. I calculated it for a Japanese dictionary with 200000 entries. They would need approximately 60000 html files. This is already close to the maximum number of files in the 32-bit zip format. I do not know whether the Kobo can handle 64-bit zip format. I also do not know how this huge number of files would influence the performance.

As for the Japanese dictionary, the Kobo takes a different approach, which however does no work very well (cf. this post).

ShellShock · 11-18-2012, 01:30 PM

Today I tried everything I could think of to get links working within a dictionary, without any success. Because the html pages are compressed, normal hrefs will not work, and Kobo do not seem to have added any extension to make them work with compressed files; this would have been a relatively simple thing for them to do, and would have enhanced their own dictionaries as well as any custom ones we build.

tshering · 11-18-2012, 02:24 PM

Links would really be great. I would like to use them for letting the user disambiguate in cases of homonymity and similar cases.
Did you try which html tags and properties can be used (and work)?

ShellShock · 11-18-2012, 04:01 PM

Most tags work fine. The following definitely work: ol, ul, li, br, small, b, sup, sub, plus the support for UTF-8 is good, which means you can use the full character set. Anchor tags (a) display as a hyperlink which flashes when pressed, but obviously cannot find the link, which is usually in a different gzipped html file. I tried lots of things in the href, but it is not going to work because the linked file is not plain html, but gzipped inside a zip. I even tried not gzipping the html (just plain html in the zip), but it did not work.

tshering · 11-18-2012, 04:29 PM

Thank you for the information. I tried to change the font size by CSS inline style, but it did not work.

ShellShock · 11-18-2012, 04:33 PM

I have just tried an href to an plain html file in the .kobo/dict folder, and that did not work either, e..g., inside my dictionary I had

Code:

<a href="fi.html#first">

where fi.html was a plain html file in .kobo.dict. Do you know the absolute pathname for the .kobo/dict directory within the device's file system?

11-15-2012, 05:05 PM	#65
ShellShock Wizard Posts: 1,178 Karma: 2431850 Join Date: Sep 2008 Device: IPad Mini 2 Retina	I have solved the mystery of why word lookup in my custom dictionary was not working - I could not look up any words shorter than 4 characters. Also marisa-lookup always returned -1 no matter what. I had to get up close and personal with Marisa, the naughty thing, and put her through a debugger. This showed that all my keys had a carriage-return on the end. Ah ha - Marisa only plays nice with Unix, the hussy! After stripping all my carriage-returns (leaving just newlines) from my index.txt file, Marisa was much happier, and my new dictionary is now fully working. I have also discovered that the html input files can have multiple <p> elements per <w> element, if you have multiple definitions for the same word, e.g., Code: <w> <p><a name="a"/><b>a</b>, the indefinite article</p> <p><a name="a"/><b>a</b>, ä, a dialectic corruption of <i>he</i> or <i>she</i></p> </w> On the Kobo, the multiple definitions will be displayed on the same page, with a blank line between them. The Chambers dictionary I used as my source only has 33,000 definitions (but I like them!), so I am going to look for something a bit more substantial to convert to the Kobo.

11-18-2012, 04:01 PM	#73
ShellShock Wizard Posts: 1,178 Karma: 2431850 Join Date: Sep 2008 Device: IPad Mini 2 Retina	Most tags work fine. The following definitely work: ol, ul, li, br, small, b, sup, sub, plus the support for UTF-8 is good, which means you can use the full character set. Anchor tags (a) display as a hyperlink which flashes when pressed, but obviously cannot find the link, which is usually in a different gzipped html file. I tried lots of things in the href, but it is not going to work because the linked file is not plain html, but gzipped inside a zip. I even tried not gzipping the html (just plain html in the zip), but it did not work. Last edited by ShellShock; 11-18-2012 at 04:15 PM.

11-18-2012, 04:33 PM	#75
ShellShock Wizard Posts: 1,178 Karma: 2431850 Join Date: Sep 2008 Device: IPad Mini 2 Retina	I have just tried an href to an plain html file in the .kobo/dict folder, and that did not work either, e..g., inside my dictionary I had Code: <a href="fi.html#first"> where fi.html was a plain html file in .kobo.dict. Do you know the absolute pathname for the .kobo/dict directory within the device's file system?

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
What's file format of dictionary	mnjkl	Kobo Reader	2	12-12-2011 08:48 AM
Dictionary format	jgray	Sony Reader	1	10-25-2010 09:52 AM
English Thesaurus in the dictionary format	osnova	Amazon Kindle	14	12-12-2009 06:42 PM
Dictionary: what version? can it be in firmware?	jedix	Sony Reader Dev Corner	7	12-05-2008 12:00 PM
Webster dictionary in DEPReader format	abigail	Reading and Management	0	08-10-2005 08:00 AM

11-13-2012, 05:44 PM	#61
tshering Wizard Posts: 3,489 Karma: 2914715 Join Date: Jun 2012 Device: kobo touch	Thank you, ShellShock!

11-14-2012, 05:08 PM	#62
ShellShock Wizard Posts: 1,178 Karma: 2431850 Join Date: Sep 2008 Device: IPad Mini 2 Retina	Just a quick update - I have done a first pass conversion of the Chambers Twentieth century dictionary circa 1913 on Project Gutenberg so it now runs on my Glo. It is still very buggy: can't find 1 or 2 character words, links in definitions to other words do not work, lookup for e.g., "hand" complains about not finding it and suggests the closest match as "Hand", still claims the Source is "Merriam-Wedtser" etc etc. On the last point - I had a poke around the Sqlite db on the reader; there is a dictionary table on there, but it doesn't seem to hold all the information, e.g., no "Source". I tried adding a new record for my custom dictionary, but the reader did not recognise it, so I am still replacing the existing dicthtml.zip with my custom dictionary.

11-15-2012, 05:20 PM	#66
Terisa de morgan Grand Sorcerer Posts: 6,599 Karma: 12595249 Join Date: Jun 2009 Location: Madrid, Spain Device: Kobo Clara/Aura One/Forma,XiaoMI 5, iPad, Huawei MediaPad, YotaPhone 2	Only a "small" comment: Marisa is a real Spanish nichname

11-17-2012, 03:16 PM	#68
shutramp Junior Member Posts: 4 Karma: 10 Join Date: Nov 2012 Device: kobo glo	Would be possible to make one for chinese as well?

11-17-2012, 05:27 PM	#69
AlPe Digital Amanuensis Posts: 727 Karma: 1446357 Join Date: Dec 2011 Location: Turin, Italy Device: Several eReaders and tablets	Well, I guess that if it works for Japanese should work for Chinese as well --- but I am not a Unicode Master, so do not take my word for granted.

11-18-2012, 01:30 PM	#71
ShellShock Wizard Posts: 1,178 Karma: 2431850 Join Date: Sep 2008 Device: IPad Mini 2 Retina	Today I tried everything I could think of to get links working within a dictionary, without any success. Because the html pages are compressed, normal hrefs will not work, and Kobo do not seem to have added any extension to make them work with compressed files; this would have been a relatively simple thing for them to do, and would have enhanced their own dictionaries as well as any custom ones we build.

11-18-2012, 02:24 PM	#72
tshering Wizard Posts: 3,489 Karma: 2914715 Join Date: Jun 2012 Device: kobo touch	Links would really be great. I would like to use them for letting the user disambiguate in cases of homonymity and similar cases. Did you try which html tags and properties can be used (and work)?

11-18-2012, 04:29 PM	#74
tshering Wizard Posts: 3,489 Karma: 2914715 Join Date: Jun 2012 Device: kobo touch	Thank you for the information. I tried to change the font size by CSS inline style, but it did not work.

Advert

Advert