New dictionary format of firmware 2.14

[tshering]

I am not sure what exactly is the "idea" behind the 11.html. As, e.g., the case of "o'clock" shows (it works in 11.html and in o'.html) the rules are not so clear-cut. To my mind the rule should be: Put into 11.html all cases that otherwise would result in an invalid filename (on any? OS), period not being accepted as part of a filename proper. I did however not test whether the search engine finds for instance "3rd" in 3r.html.

[tshering]

Originally Posted by AlPe

... every key NOT starting as [a-z][a-z] (when lowercased) will go to 11.html.

This sends all expressions of some languages to 11.html.

[AlPe]

Originally Posted by tshering

This sends all expressions of some languages to 11.html.

Do not take that expression too literally: it was a short-hand for "every word (lowercased) NOT starting with two letters --- in the range a-z. My code actually generates other files other than 11.html

[tshering]

To be more clear, I am thinking of languages that do not use Latin script.

[AlPe]

I tried this dictionary:

<?xml version = "1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE document SYSTEM "dictionary.dtd"> 
<dictionary>
<entry>
 <key>Würstel</key>
 <def>Definition of <i>Würstel</i>, with upper case letter.</def>
</entry>
<entry>
 <key>würstel</key>
 <def>Definition of <i>würstel</i>, with lower case letter.</def>
</entry>
<entry>
 <key>1826</key>
 <def>Test of <i>1826</i> one-eight-two-six.</def>
</entry>
<entry>
 <key>è</key>
 <def>"è" is the Italian equivalent of "is".</def>
</entry>
<entry>
 <key>Leopardi</key>
 <def><i>Leopardi</i> is the name of a famous Italian poet.</def>
</entry>
<entry>
 <key>leopardi</key>
 <def><i>leopardi</i> means "leopards" in Italian.</def>
</entry>
<entry>
 <key>égloga</key>
 <def>Type of ancient poem.</def>
</entry>
</dictionary>

My code generates le.html (Leopardi and leopardi) and 11.html (all the other words). All the keys are present in words, and they are recognized by the search function on the kobo (tested this by searching for them in search box).

Apparently, searching for "1826" retrieves its definition, while "égloga" or " würstel" (they went into 11.html) does not --- this could mean that the Kobo expects them to be in ég.html and wü.html files. So far so good, but I am puzzled by the fact that also "è" does not retrieve its definition, albeit being in 11.html.

[AlPe]

Originally Posted by tshering

To be more clear, I am thinking of languages that do not use Latin script.

Yes, that's sure. How should this be dealt with?

[tshering]

Originally Posted by AlPe

So far so good, but I am puzzled by the fact that also "è" does not retrieve its definition, albeit being in 11.html.

The Kobo expects it to be in éa.html

[tshering]

Originally Posted by AlPe

Yes, that's sure. How should this be dealt with?

I think any character beyond 127 is acceptable. For instance all Japanese characters work. We only have to find out which characters from 0-127 are not acceptable.

[AlPe]

Originally Posted by tshering

The Kobo expects it to be in éa.html

You are right, I tried and I confirm it.

[AlPe]

Originally Posted by tshering

I think any character beyond 127 is acceptable. For instance all Japanese characters work. We only have to find out which characters from 0-127 are not acceptable.

Seems legit. I will try to check tomorrow. Thanks for the lead.

[gouni]

@Alpe, you want that I you send my couple of letter French dictionary for your test?

[AlPe]

Yes, please, especially if you have already experimented on which keywords should go to which XX.html file.

[AlPe]

Ok, I have performed a test of ASCII (0-127) characters, except control characters, and it seems that you can move all keywords containing a non-letter (defined as [A-Za-z]) in the first two characters into 11.html.

I attach the XML dictionary I used for testing, plus the "Italian" Kobo dictionary compiled by Penelope from the XML dictionary. (Note: download the XML file and open it with a text editor, since it might not display properly in a browser, since it has raw "< > &". Penelope does not use a DOM parsers, hence it allows those characters to be unescaped.)

I updated the Google Code source code of Penelope, to reflect tshering's suggestion (thanks!) about 1-character keywords.

[gouni]

@Alpe I you send a mail.

[tshering]

Originally Posted by AlPe

Ok, I have performed a test of ASCII (0-127) characters, except control characters, and it seems that you can move all keywords containing a non-letter (defined as [A-Za-z]) in the first two characters into 11.html.

In order to work also with languages that use other scripts than Latin script, e.g. Greek, it would be preferable that your script sends to 11.html only those keywords that must be send to 11.html, rather that all those keywords that can be send to 11.htrml. This should be only a small change in your script but have a great effect on the usability.