Firmware 4.24.15672

[davidfor]

Originally Posted by geek1011

I looked into this a bit more, and found that this won't be possible.

prefix_exceptions is somewhat of a misnomer, since it doesn't actually make exceptions for prefixes. Instead, it should be called word_redirects, since it just changes the word being looked up to another if it matches exactly. The target file must still have a variant/word matching the new one, and the original file won't be looked in at all.

This also means that there's already a bug in prefix_exceptions, albeit the inverse of the reason why prefix_exceptions was created. Previously, with v2 dictionaries, variants with a different headword prefix wouldn't be found (I worked around this in dictutil by duplicating the entries). Now, if you have a headword named after a redirected variant, it won't be found. For example, with the previous v2 behaviour, the entry for go/went would need to be duplicated into go.html and we.html, and you could also have another unique definition titled went in we.html. With the new v3 behaviour, you can just define it go/went in go.html and add a redirect entry like "went\tgo" to redirect it. But, this is where the new bug happens. Now, if you had a second entry in we.html named "went" (remember that Kobo dictionaries support multiple entries for a word), it won't be found since the words was redirected to "go". I can work around this bug by duplicating the headwords into the redirected files...which is just the counterpart to my previous workaround.

This bug could be fixed on Kobo's side by also looking up the original search term in addition to the redirected one.

Once Kobo releases the new dictionaries, I'm going to see if they have this bug (note: the current v2 dictionaries have the old bug with variant prefixes which are different than the headword prefix).

@davidfor, do you know anything about this behaviour/bug?

Only what I read above. To be honest, while I use the dictionaries frequently, I don't play with them much. I have never bothered trying to install custom dictionaries except to test some of the statements made here about it. If I don't find a word in the dictionary, I just use the web or Wikipedia search. Or mark it to lookup later.

Reading the above, I'm not actually sure what to expect. From this, I think you are saying that there are definitions for both "go" and "went" and a redirect from "went" to "go". I think that you are then saying if I look up "go" I should see the definitions of the words "go". And if I look up "went", I should see the redirect to "go" plus whatever is there for "went". Is that correct? But we are only seeing the definitions for "go" and not the other definitions for "went".

Is that correct? Do you have any other examples? Or maybe a test dictionary that just has a few examples of this.

[geek1011]

Originally Posted by davidfor

Reading the above, I'm not actually sure what to expect. From this, I think you are saying that there are definitions for both "go" and "went" and a redirect from "went" to "go". I think that you are then saying if I look up "go" I should see the definitions of the words "go". And if I look up "went", I should see the redirect to "go" plus whatever is there for "went". Is that correct? But we are only seeing the definitions for "go" and not the other definitions for "went".

Is that correct? Do you have any other examples? Or maybe a test dictionary that just has a few examples of this.

That's almost correct. One correction: if you look up "went", you should see the "went" definitions, but not the "go" ones (since exact matches take priority over variants). Note that this is an test case I came up with, not a real example from the official English dictionary.

---

A simple example:

bug.df

Spoiler:

@ go
& went
go & went #1

@ go
& went
go & went #2

@ went
went #1

@ went
went #2

@ test1
& test2
test1 & test2

@ test3
& test2

test3 & test2

See here for the dictfile format. TLDR: @ starts a new entry, & adds a variant, and the lines after it are the body in markdown.

dictgen --v3-prefix-exceptions --output dicthtml-bug-v3.zip bug.df
- This will generate a v3 dictionary from the dictutil dictfile.
- Note that the --v3-prefix-exceptions option hasn't been pushed to GitHub yet, but I can give you the code if you want.
- dictutil will add a prefix exception for the latest occurrence of each variant back to the headword if the prefixes do not match.

dictutil unpack dicthtml-bug-v3.zip && zip -r dicthtml-bug-v3.zip dicthtml-bug-v3

dictgen --output dicthtml-bug-v2.zip bug.df
- This will generate a v2 dictionary from the dictutil dictfile.
- This includes the workaround I implemented in dictutil where variants which do not match the headword prefix will be duplicated into the html file for that variant's prefix.

dictutil unpack dicthtml-bug-v2.zip && zip -r dicthtml-bug-v2.zip dicthtml-bug-v3

Copy dicthtml-bug-v{2,3}.zip to .kobo/custom-dict on 15672+, and try searching "go", "went", "test1", "test2", and "test3" for each dictionary and compare the output.

The output for "test1" and "test3" are equivalent and correct for both versions (each respective word will display on its own).

The output for "test2" is also equivalent and correct for both versions (both "test1" and "test3" will be displayed). Note that if a prefix_exception was added mapping "test2" to "test3", only "test3" will be displayed even though the original word was "test2" and "test1" has a variant of "test2" (because the word from the exception is treated as if it was the original search query).

The output for "go" is equivalent and correct for both versions (the two entries for the word will display consecutively).

The output for "went" is correct on v2, but not on v3. On v2, it will display the two entries for "went", but not the "go" entries (which is expected since an exact match takes precedence over a variant). On v3, it will display the "go" entries only due to the redirect.

There are a few other things related to this, but I didn't include them for the sake of brevity.

Note that most of these issues could be considered the fault of the dictionary creator, as it's possible to work around them by not generating prefix_exceptions for cases like this (since they would be treated correctly without them). The remaining cases can be worked around with a combination of duplicating entries, but that's basically eliminating any advantages of having prefix_exceptions.

Also note that this behaviour would make more sense if we consider prefix_exceptions to be a list of redirects rather than exceptions. Personally, I think this feature would have been more useful as a list of additional prefixes to search for a headword in, but it's too late for that to be changed (and has limited benefit for Kobo's own use case).

A possibility for changing the behaviour of this would be to query the original html file too. Then, the most specific match could be returned (an exact headword match in the original, else an exact variant match in the new one, etc). Alternatively, entries from both could be returned, but this would not be fully backwards-compatible with the v2 behaviour (specifically with entries duplicated into multiple html files as a workaround for the actual bug with variants in v2).

After some thought, I don't think Kobo's official dictionaries are likely to run into this issue unless they either:
- Make redirects for all variants, even if they are in the same file.
- Redirect to a variant which isn't a headword anywhere.
- A few other small possibilities.

The behaviour I'm describing here is more of an inconsistency between v2/v3 than a bug. I don't expect it to be fixed, and I wouldn't even recommend that at this point. I am mainly looking at this so I am aware of the handling of edge cases.

I will be able to take a more definite stance on this once I see the new official dictionaries in October and how they make use of prefix_exceptions. I won't be releasing the dictutil v3 dictionary support until I see the official dictionaries so I can match their behaviour as closely as possible.

Edit: According to @davidfor, the behaviour has been improved in 15676. I'll look at those changes later this week and post in the other thread.

[geek1011]

Originally Posted by rtiangha

I just discovered that you could also make a file with the format of dicthtml-xx-xx-yyyyy.zip where yyyyy is any string (so for example, dicthtml-en-en-wordnet.zip or dicthtml-en-en-gcide.zip, etc.) and they'll also show up in the menu (in this case, as English - English (Custom)), which is really cool. They'll all have the same name in the menu though and I don't know if there's a limit to the number you can have installed at the same time, but it would appear that you can now have multiple English (or any) language dictionaries without modifying locales, which is great if you don't want to have to merge multiple dictionaries together under the old way. But I only tested with English as the source language; haven't tried others yet.

You can actually take it a step further (I just didn't include them in my list in that post since it wasn't relevant to my point there). You can actually name it dicthtml*.zip, where * is literally any valid sequence of characters which are not whitespace. It doesn't have to be a valid locale or locale-pair anymore.

[Gone]

Originally Posted by davidfor

Hopefully so. From the 1st of October, most of the dictionaries are changing. Some are changing suppliers (the English dictionary will come from Oxford University Press) and some are changing versions (the Dutch dictionaries appear to be a different version). From comments elsewhere, the English dictionary is a much better dictionary. Hopefully the same can be said for the others.

The changes for custom dictionary handling will simplify adding or replacing the dictionaries.

At this point, I would suggest removing the custom dictionaries and trying the new ones. If you don't like them, you can always change back.

The french one is the same, Micro Robert 2013, which is a very small dictionnary containing mostly known words.

I still wish Kobo would at least let us buy a real dictionnary.

[Uncle Robin]

Originally Posted by Claude_C

The french one is the same, Micro Robert 2013, which is a very small dictionnary containing mostly known words.

I still wish Kobo would at least let us buy a real dictionnary.

Perhaps French will be getting one of the new ones on October 1st.

[davidfor]

Originally Posted by Claude_C

The french one is the same, Micro Robert 2013, which is a very small dictionnary containing mostly known words.

The updates to the dictionaries aren't supposed to be available until the 1st. I wouldn't expect you to be seeing any differences. But. I was told that the French dictionary would remain as Mirriam-Webster. But, maybe they meant that it wasn't changing. Or their arrangement for it is through Mirriam-Webster.

I believe at least part of the reason for the dictionary changes is that licenses have expired. It could be that the French dictionary was part of a different deal. Or that they couldn't find a better supplier

I still wish Kobo would at least let us buy a real dictionnary.

I have always liked that idea. I don't know if it is something they don't want to do, or there are issue with implementing it. I can see the legal issues. And of course, how much would people be willing to pay? That might make it not worth it to them. Or maybe just not enough people want it.

[Poppaea]

Originally Posted by davidfor

I have always liked that idea. I don't know if it is something they don't want to do, or there are issue with implementing it. I can see the legal issues. And of course, how much would people be willing to pay? That might make it not worth it to them. Or maybe just not enough people want it.

I would like to be able to buy one, too! Very much so. The current English-German dictionary is atrocious.

For example if I want to look up "maim" it says:

"No match for "maim", closest match was "maid".
maidn

• Dienstmädchen

Which isn't bad at all, given I almost never get even a wrong answer. Most of the time it just says it can't find anything, like for "taught". It is a rare thing for me not to know a word, the ones above I all knew, but if I need to look up on, their dictionary is utterly useless. Might as well have none.

The Tolino OTOH has a very fine dictionary, which has always had what I needed. Same is true for the Sony, theirs was even better than the Tolinos. So it is possible to implement something good. It is just not possible for stingy Kobo.

I'd so much like to spend money on a good dictionary, I could implement easily myself!

[John Hopper]

Yes, it would be great to be able to import easily new dictionaries. So, for example, for my Kindles, I bought a Latin dictionary and added it to the dictionaries folder, which is really useful. Would be really useful for Kobo as well.

Sent from my SM-N976B using Tapatalk

[Poppaea]

Originally Posted by John Hopper

Yes, it would be great to be able to import easily new dictionaries. So, for example, for my Kindles, I bought a Latin dictionary and added it to the dictionaries folder, which is really useful. Would be really useful for Kobo as well.

Sent from my SM-N976B using Tapatalk

You read Latin? I think What are you reading?

Last time I read Latin was when I was in my early twenties. Tacitus, because I could and I wanted to make use of years of pointlessly learing it and because it would seriously impress everyone. Then I met someone who read Greek philosophers in the original and quietly dropped it.

[toancv]

Originally Posted by geek1011

I looked into this a bit more, and found that this won't be possible.

prefix_exceptions is somewhat of a misnomer, since it doesn't actually make exceptions for prefixes. Instead, it should be called word_redirects, since it just changes the word being looked up to another if it matches exactly. The target file must still have a variant/word matching the new one, and the original file won't be looked in at all.

This also means that there's already a bug in prefix_exceptions, albeit the inverse of the reason why prefix_exceptions was created. Previously, with v2 dictionaries, variants with a different headword prefix wouldn't be found (I worked around this in dictutil by duplicating the entries). Now, if you have a headword named after a redirected variant, it won't be found. For example, with the previous v2 behaviour, the entry for go/went would need to be duplicated into go.html and we.html, and you could also have another unique definition titled went in we.html. With the new v3 behaviour, you can just define it go/went in go.html and add a redirect entry like "went\tgo" to redirect it. But, this is where the new bug happens. Now, if you had a second entry in we.html named "went" (remember that Kobo dictionaries support multiple entries for a word), it won't be found since the words was redirected to "go". I can work around this bug by duplicating the headwords into the redirected files...which is just the counterpart to my previous workaround.

This bug could be fixed on Kobo's side by also looking up the original search term in addition to the redirected one.

Once Kobo releases the new dictionaries, I'm going to see if they have this bug (note: the current v2 dictionaries have the old bug with variant prefixes which are different than the headword prefix).

@davidfor, do you know anything about this behaviour/bug?

Edit: I just thought of a way to make the chinese dictionary hack possible even with the behaviour I discovered. I can hash each entry as a hidden variant, then add redirects for the real headwords to that hash (essentially limiting the number of prefixes to 2 ASCII alphanumeric characters).

It is strange that my custom Japanese dictionary does not work any more while the Chinese one still works fine on the new FW 4.24. Both dictionaries created by your dictgen and are still working good on FW 4.23. This is really strange, maybe Kobo want to deal something special with Japanese dictionaries?

[John Hopper]

Originally Posted by Poppaea

You read Latin? I think What are you reading?



Last time I read Latin was when I was in my early twenties. Tacitus, because I could and I wanted to make use of years of pointlessly learing it and because it would seriously impress everyone. Then I met someone who read Greek philosophers in the original and quietly dropped it.

At the risk of losing your love I must admit I don't read whole Latin texts, though I did study it to A level many years ago. But it is useful to consult for references in literature and history books I read.

Sent from my SM-N976B using Tapatalk

[geek1011]

Originally Posted by toancv

It is strange that my custom Japanese dictionary does not work any more while the Chinese one still works fine on the new FW 4.24. ... This is really strange, maybe Kobo want to deal something special with Japanese dictionaries?

Yes, those are handled completely differently (but only if the dictionary locale is Japanese). What changed in particular?

Both dictionaries created by your dictgen and are still working good on FW 4.23.

Thanks for the confirmation, it's as I expected (in theory, based on the firmware, v2 should be fully compatible with v3).

[geek1011]

Originally Posted by John Hopper

Yes, it would be great to be able to import easily new dictionaries. So, for example, for my Kindles, I bought a Latin dictionary and added it to the dictionaries folder, which is really useful. Would be really useful for Kobo as well.

If you can find the dictionary in an alternate format, you can use PyGlossary to convert it to a dictutil dictfile, then to a Kobo dictionary. I wouldn't use the built-in PyGlossary Kobo output directly just yet, as it's not fully mature (it's progressing nicely, though).

[ZodWallop]

Originally Posted by the.Mtn.Man

How can we can tell if we have the new Oxford dictionaries?

I keep checking the thread hoping to see an answer to this.

[davidfor]

Originally Posted by ZodWallop

I keep checking the thread hoping to see an answer to this.

By looking at the date? The release notes explicitly state "as of October 1, 2020". But, if you have the new dictionaries, it will have a copyright statement the end of the definition.