Best custom dictionary? (English)

[rtiangha]

Originally Posted by skybook

Thank you for the file! Yeah, I must have done something wrong, because my version of this file is only 50 458 624 bytes, vs yours which is 95 195 136 bytes :/

No, I think you did it right; when I just ran it, I got 49,789,808.

So I don't know. I might have used different source files or did something more to them (I don't think it was just --flatten-synonyms; it didn't make the file sizes grow enough.). This was all last year and I didn't take notes and deleted my build directories to save space. My bash history doesn't go back that far either. Oh, well. At least you have my end result. Let me know what you think.

[geek1011]

Note that dictutil is not intended to be as comprehensive as Penelope. It will just be for generating, unpacking, packing, testing (maybe), and installing Kobo dictionaries. Converting between formats (other than dicthtml and dictutil's own format) is out of scope.

For people using Penelope, be aware that prefix generation is incorrect in some cases, like when words have special characters after the first two characters, or for some accented letters.

[rtiangha]

Originally Posted by geek1011

For people using Penelope, be aware that prefix generation is incorrect in some cases, like when words have special characters after the first two characters, or for some accented letters.

How do you work around/fix that, if it's even possible? Last year, I was trying to create or combine some Japanese dictionaries, and while individual dictionaries would work, combining them sometimes would and sometimes wouldn't depending on the combination and I couldn't find a pattern. I think I wasted a month trying to figure it out, lol. But since there are so many Japanese characters to keep track of, a prefix problem sounds like a plausible cause. I know there are some penelope command line options that deal with prefix, but there aren't any usage examples in the help text so I don't know how to use them.

[geek1011]

Originally Posted by rtiangha

How do you work around/fix that, if it's even possible? Last year, I was trying to create or combine some Japanese dictionaries, and while individual dictionaries would work, combining them sometimes would and sometimes wouldn't depending on the combination and I couldn't find a pattern. I think I wasted a month trying to figure it out, lol. But since there are so many Japanese characters to keep track of, a prefix problem sounds like a plausible cause. I know there are some penelope command line options that deal with prefix, but there aren't any usage examples in the help text so I don't know how to use them.

It's a problem in the code itself (I figured out how it works the same way I figure out patches (i.e. disassembly), unlike how the dictionary stuff was original done many years ago). To fix it, you need to change kobo_prefix.py (or prefix_kobo.py, I don't remember which) to consider only the first 2 chars when determining whether to use 11.html, you need to consider if a character is in the Unicode letters class rather than just ASCII, and you might need to add a .replace(" ", "a").

I'll publish my notes on my other findings around the same time I finish dictutil.

[rtiangha]

Originally Posted by geek1011

It's a problem in the code itself (I figured out how it works the same way I figure out patches (i.e. disassembly), unlike how the dictionary stuff was original done many years ago).

OK, my Python is weak, but it would be nice to fix this if possible, especially if more is known about the file format now than when penelope was originally developed. So while I wait for your notes, I'm going to try to reason this out without trying to code trace everything (which is something I don't really want to do, lol) and maybe even someone with the skills might be able to fix things if I don't get around to it or can't figure it out in time.

To fix it, you need to change kobo_prefix.py (or prefix_kobo.py, I don't remember which)

Definitely prefix_kobo.py.

to consider only the first 2 chars when determining whether to use 11.html,

I saw your reply to skybook in the other thread. So is it just enough to change this portion:

        if is_ok:
            prefix = headword[0:length]

to this:

        if is_ok:
            prefix = headword[0:2]

Or is there more to it?

you need to consider if a character is in the Unicode letters class rather than just ASCII,

So clearly, this part:

    def is_allowed(character):
        # all non-ascii (x > 127) are ok
        # all ASCII lowercase letters (97 <= x <= 122) are ok
        # everything else is not ok
        try:
            code = ord(character)
            return (code > 127) or ((code >= 97) and (code <= 122))
        except:
            pass
        return True

needs to be rewritten. I think x > 127 was intended to take care of things like Japanese, Chinese, Arabic, etc. characters, but I'm guessing that there's stuff in there that needs to be ignored too (like emojis and other symbols)? Or are there other things to be concerned about?

and you might need to add a .replace(" ", "a")..

That part I'm not sure about, but is it related to the

For entries like "copper noble", those should have gone into files like "co.html" instead of "11.html", but that's a bug in Penelope.

issue in the other thread? As in (if I'm reading the code right; remember, weak in Python) the entry gets treated as a special case because of the space (because SPACE has a ASCII/Unicode value of 64 which is outside of the range that the is_allowed() function is looking at) and so replacing all spaces with 'a' (or really, any character in the allowable range?) is needed so the code as currently written treats it as a headword with no spaces (which it is able to process correctly) instead? Or did I get that completely wrong?

Also, does dictutil handle Kobo dictionary synonyms properly? Penelope doesn't even touch it, and stuff like the Japanese dictionaries rely heavily on synonyms because the language uses three alphabets (Kanji, hiragana and katakana) and a complex word can be spelt with any combination (ex. All kanji, all hiragana, all katakana, a mix of kanji and hiragana, or even in Latin letters via romaji) and so it leans heavily on synonyms so that there aren't separate/duplicate entries for each different way to spell a word.

While many of the stock Kobo dictionaries are encrypted, the Japanese ones currently aren't, if you'd like to take a peek at how they use synonyms (I actually managed to convert the Progressive EN-JA kobo dictionary to Stardict XML format, which considering I don't know regular expressions or XSLT at all (I did it with a lot of find/replace in Notepad++, lol), I was amazed that it even worked and immediately sought to merge it with entries from the open-source JMDict project, albeit an older version (using --flatten-synonyms, of course); in fact, part of my struggle was in creating an updated JMDict version with 2019 data because my Kobo wouldn't recognize what I made no matter what I did, which in hindsight, might probably be due to " " appearing in some headwords. So yeah, it'd be nice to fix penelope if possible).

[geek1011]

Originally Posted by rtiangha

OK, my Python is weak, but it would be nice to fix this if possible, especially if more is known about the file format now than when penelope was originally developed. So while I wait for your notes, I'm going to try to reason this out without trying to code trace everything (which is something I don't really want to do, lol) and maybe even someone with the skills might be able to fix things if I don't get around to it or can't figure it out in time.



Definitely prefix_kobo.py.



I saw your reply to skybook in the other thread. So is it just enough to change this portion:

        if is_ok:
            prefix = headword[0:length]

to this:

        if is_ok:
            prefix = headword[0:2]

Or is there more to it?



So clearly, this part:

    def is_allowed(character):
        # all non-ascii (x > 127) are ok
        # all ASCII lowercase letters (97 <= x <= 122) are ok
        # everything else is not ok
        try:
            code = ord(character)
            return (code > 127) or ((code >= 97) and (code <= 122))
        except:
            pass
        return True

needs to be rewritten. I think x > 127 was intended to take care of things like Japanese, Chinese, Arabic, etc. characters, but I'm guessing that there's stuff in there that needs to be ignored too (like emojis and other symbols)? Or are there other things to be concerned about?



That part I'm not sure about, but is it related to the



issue in the other thread? As in (if I'm reading the code right; remember, weak in Python) the entry gets treated as a special case because of the space (because SPACE has a ASCII/Unicode value of 64 which is outside of the range that the is_allowed() function is looking at) and so replacing all spaces with 'a' (or really, any character in the allowable range?) is needed so the code as currently written treats it as a headword with no spaces (which it is able to process correctly) instead? Or did I get that completely wrong?

Yes, [:2] is correct.

And to check if it is a Unicode letter, use isalpha.

No, that last part is incorrect, as only the first two characters are considered. But, something like "a" would to into "aa.html".

You might also need to change the order of some of the checks to match. I'll give more details once I finish dictutil.

Also, does dictutil handle Kobo dictionary synonyms properly? Penelope doesn't even touch it, and stuff like the Japanese dictionaries rely heavily on synonyms because the language uses three alphabets (Kanji, hiragana and katakana) and a complex word can be spelt with any combination (ex. All kanji, all hiragana, all katakana, a mix of kanji and hiragana, or even in Latin letters via romaji) and so it leans heavily on synonyms so that there aren't separate/duplicate entries for each different way to spell a word.

Yep, that's one of the best parts! So, there are a few things I figured out. Firstly, if more than one entry matches, Kobo will merge it in the order it appears in the *same* file (i.e. not files for other prefixes) (I need to check this further). Secondly, variants MUST be trimmed and lowercased in the HTML files, regardless of the casing in the index or in the actual word. This is because unlike usual headword matches (original, lowercased, uppercased, and lowercased with first letter uppercased; all by prefix), variants are only matched against the lowercased version. Thirdly, the entire entry for variants must be duplicates into all matching prefixes (this is a bug in the official dictionaries too).[/QUOTE]

While many of the stock Kobo dictionaries are encrypted, the Japanese ones currently aren't, if you'd like to take a peek at how they use synonyms (I actually managed to convert the Progressive EN-JA kobo dictionary to Stardict XML format, which considering I don't know regular expressions or XSLT at all (I did it with a lot of find/replace in Notepad++, lol), I was amazed that it even worked and immediately sought to merge it with entries from the open-source JMDict project, albeit an older version (using --flatten-synonyms, of course); in fact, part of my struggle was in creating an updated JMDict version with 2019 data because my Kobo wouldn't recognize what I made no matter what I did, which in hindsight, might probably be due to " " appearing in some headwords. So yeah, it'd be nice to fix penelope if possible).

I've checked against all the Kobo dictionaries, but don't ask me how. It's not really feasible to add synonyms to Penelope without either hacky stuff (putting them into the definition field), or a major restructure.

[rtiangha]

Originally Posted by geek1011

You might also need to change the order of some of the checks to match. I'll give more details once I finish dictutil.

Ok, I'l wait for your notes.

I've checked against all the Kobo dictionaries, but don't ask me how.

Um, if you're saying what I think you're saying and you figured it out the way I think you figured it out, then I'm definitely not letting my Kobo get online in case the same information that protects the dictionaries is also used to protect communications, lol.

It's not really feasible to add synonyms to Penelope without either hacky stuff (putting them into the definition field), or a major restructure.

When thinking about general algorithms at high-level, I did consider embedding things in the definition field, although I wasn't sure how to handle merged definitions from other dictionaries that might have a different set of synonyms for the same word (figuring out how to say it in Python however would be a different challenge for me).

But for just going Stardict->Kobo, I almost think it'd be easier to decompile the dictionary into Stardict XML format and then use XSLT to transform the definitions to Kobo format. And I think you'd be able to go in the opposite direction using that method as well. Or maybe something like that ends up being a totally different utility instead.

[geek1011]

Originally Posted by rtiangha

Um, if you're saying what I think you're saying and you figured it out the way I think you figured it out, then I'm definitely not letting my Kobo get online in case the same information that protects the dictionaries is also used to protect communications, lol.

No comment, other than if you think I've cracked SSL, that's certainly not the case (dict encryption and HTTPS are protected differently).

When thinking about general algorithms at high-level, I did consider embedding things in the definition field, although I wasn't sure how to handle merged definitions from other dictionaries that might have a different set of synonyms for the same word (figuring out how to say it in Python however would be a different challenge for me).

Yes, that would be a limitation of that method and the way Penelope processes stuff.

But for just going Stardict->Kobo, I almost think it'd be easier to decompile the dictionary into Stardict XML format and then use XSLT to transform the definitions to Kobo format. And I think you'd be able to go in the opposite direction using that method as well. Or maybe something like that ends up being a totally different utility instead.

Or, you could use XSLT to transform it into my own (future) format. To merge dictionaries in my format, all you'll need to do is append the files (most likely, I'll need to test a few details, such as how variants are matched). More on this later.

[BaronMoe]

Thanks for the detailed answers guys. So I am seeing two viable candidates here.

rtiangha's 90mb version and Owl's 55 mb. Going off file sizes, the most thorough is rtiangha's?

Ill add both for the time being and report back on their performance.

[Semwize]

Originally Posted by BaronMoe

Going off file sizes, the most thorough is rtiangha's?

Ill add both for the time being and report back on their performance.

If combine dictionaries, there will be a lot of repetitions (identical words and similar descriptions). Therefore, the larger the size of the dictionary is not the fact that it is better. It is necessary to check the result. Watch and search for words.

[koboy]

Originally Posted by rtiangha

No, I think you did it right; when I just ran it, I got 49,789,808.

So I don't know. I might have used different source files or did something more to them (I don't think it was just --flatten-synonyms; it didn't make the file sizes grow enough.). This was all last year and I didn't take notes and deleted my build directories to save space. My bash history doesn't go back that far either. Oh, well. At least you have my end result. Let me know what you think.

does this mean the 90.8MB file size can be made smaller/down to 50MB?

best wishes koboy

[Alanon]

I'm a bit late to the party, but I'll chime in, as I'm a bit of a dictionary nerd and quite like fiddling with them on my Kobo.

I believe that Kindles use the ODE (Oxford Dictionary of English), which is not the OED (Oxford English Dictionary), nor is it based on that venerable enterprise. It's a modern single-volume dictionary that OUP produced twenty-odd years ago. The dictionary that is based on the OED is called the Shorter Oxford English Dictionary (SOED) and, unlike the OED, is much more frequently updated in print form and is currently in its sixth edition.

That being said, it is possible to get the (more or less) complete OED on a Kobo, and it works just fine speed-wise, the size and scope don't seem to be an issue. That's certainly the most comprehensive dictionary available, though it's not exactly the most user-friendly. I removed the quotations from the one I converted, and it can still get a bit prickly to navigate through the data. Some references can still have dozens of Kobo popup pages, which is far from ideal. Perhaps it's due to the way in which I made it, but the SOED is a much more practical solution in most situations. I haven't used the ODE for a while, but I remember the SOED being similar to it, yet more comprehensive due to the OED parentage.

Another middle-ground would be the equally venerable Webster's Third International. Not as comprehensive as the OED, but the entries can be a good deal shorter. For all its advantages and somewhat legendary cultural status, Webster's Third suffers from occasional stodginess. If you'd like a chuckle, look up Webster's definition of "door". Though most of its quirks were expunged from the modern, online-only edition, the spirit of Webster's remains in those defining moments.

Outside of these, I should probably mention American Heritage Dictionary and Random House Webster's Unabridged (not related to Webster's Third). I have a habit of not using them, though they are very robust, modern dictionaries with no-nonsense definitions, usually well-formulated and clear, if a bit short on examples. For the literary-minded, a good choice is also Webster's 1912 dictionary - though it took me a while to track down a version that didn't omit the best bits - the extensive quotations. For a quick look-up, the best path is probably something like Roget's 21st Century Thesaurus, that simply lists out synonyms rather intelligently.

Most of these I have on me, either in a format that Kobo will read or in a digital form that might be converted for Kobo, so feel free to hit me up for anything.

[NiLuJe]

@Alanon: I am mildly intrigued . Which source did you use for the SOED, because, AFAIK, none of the digital offerings available are in a sane format (i.e., I'm only aware of the awful Windows CD-ROM versions).

[geek1011]

For anyone interested, I've made a mingw-w64 build of marisa-trie for Windows. I've attached it to this post.

P.S. dictutil is almost ready, I just haven't been working on it for the last week and a bit.

[jackie_w]

Originally Posted by geek1011

For anyone interested, I've made a mingw-w64 build of marisa-trie for Windows. I've attached it to this post.

P.S. dictutil is almost ready, I just haven't been working on it for the last week and a bit.

Thanks for this. Do I need all the zip contents to be kept together or do the .exe's work as standalone executable files?