dictutil: Tools, documentation, and libraries related to Kobo dictionaries

[toancv]

Originally Posted by geek1011

No, it can't fix those ones. This fix is only for large dictfiles used as source content for dictgen. It's not possible to fix Kobo's support for large dictionaries, even with a patch.

They are not large dictionaries, the total number of headwords is about 100k , some english dictionaries are more headwords than that. The problems is prefix rules of Kobo dictionary make Chinese dictionary create a lot of html files (in this case is more than 65K files) which make Kobo software cannot recoginze it. Penelope method used to tried to solve this problem by giving option to reduced prefix to 1 instead of 2 letters. This option reduced number of html files for chinese dictionaries significantly (about 10K more files), but the kobo software failed to recognized such html files.

Kobo support Traditional Chinese however it don't have any built in Chinese dictionary so far. Maybe due to this problem? I don't know.

[loyukfai]

@geek1011 Do you think Kobo will revert the force-replacement behaviour in the next update?

BTW, what if I install a translation dictionary as a "normal" dictionary? Will that work as well?

Cheers.

[davidfor]

Originally Posted by loyukfai

@geek1011 Do you think Kobo will revert the force-replacement behaviour in the next update?

Only if someone can reproduce it in a repeatable and documentable way. I have seen it happen, but, it isn't consistent. And I haven't found anything in the logs to help. I don't normally sideload dictionaries, so I maybe I did something wrong (Set read-only flag and sync. Anything else?). Or I might have missed the discussion on how to break it. If so, can someone point it out.

And for the record, there are two syslogs posted in this thread. Neither show anything useful. Both show the "SyncDictionariesCommand" being run. That is what I would expect. Part of that function would be to decide if there are dictionaries to be updated, and if the read-only dictionaries shouldn't be replaced. A different log level might show more.

[loyukfai]

Noted with thanks. Is there anything I can do or provide for possible further investigation?

Cheers.

[geek1011]

Originally Posted by davidfor

Only if someone can reproduce it in a repeatable and documentable way. I have seen it happen, but, it isn't consistent. And I haven't found anything in the logs to help. I don't normally sideload dictionaries, so I maybe I did something wrong (Set read-only flag and sync. Anything else?). Or I might have missed the discussion on how to break it. If so, can someone point it out.

You're looking for geek1011/dictutil#6. Most of the other discussion happened over PMs, but the important parts are in that issue. I still have yet to figure out the exact condition checked (I just haven't got around to it yet), but the issue should be apparent if Kobo looks at their own code.

And for the record, there are two syslogs posted in this thread. Neither show anything useful. Both show the "SyncDictionariesCommand" being run. That is what I would expect. Part of that function would be to decide if there are dictionaries to be updated, and if the read-only dictionaries shouldn't be replaced. A different log level might show more.

Yes, see the issue I linked.

[davidfor]

Originally Posted by loyukfai

Noted with thanks. Is there anything I can do or provide for possible further investigation?

As I said, details of when and how it occurs are needed. Things like, do only some dictionaries get replaced? Does it only happen if there are a certain number of dictionaries? Does it happen with all syncs (manual, the scheduled sync or book syncs at other times)? Do you need to set all dictionaries to read only? What happens if you change settings. Does actually doing a dictionary lookup change things?

Otherwise, turn the logs on see if anything useful turns up. Different options will show different things. But, I haven't found a combination of options that show anything useful.

[geek1011]

Originally Posted by davidfor

Things like, do only some dictionaries get replaced?

It currently seems that for the affected people, all sideloaded dictionaries with the same name as an official one get replaced.

Does it only happen if there are a certain number of dictionaries?

The number of dictionaries doesn't have an effect.

Does it happen with all syncs (manual, the scheduled sync or book syncs at other times)?

Yes, for all syncs which update dictionaries (all of them, IIRC, but don't quote me on that one). And, the technical reason for this is that all of them build the list of dictionaries to sync using SyncDictionariesCommand:repareDownloadList, which is what should be checking if it's read-only.

Do you need to set all dictionaries to read only?

It is independent for each one.

What happens if you change settings. Does actually doing a dictionary lookup change things?

Nope.

Otherwise, turn the logs on see if anything useful turns up. Different options will show different things. But, I haven't found a combination of options that show anything useful.

Nope. Nothing logs the relevant part, which is the check for read-only, the check for the filesize, and the check for IsSynced, however that's calculated (I'll look into that if needed, but Kobo can do that themselves).

[geek1011]

Originally Posted by davidfor

Only if someone can reproduce it in a repeatable and documentable way. I have seen it happen, but, it isn't consistent. And I haven't found anything in the logs to help.

I've looked into the details of this bug, and if I've interpreted it correctly, I've narrowed it down to the file size check (see here). I might see if I can fix it with a patch later today if I have time and if the fix is simple enough.

Update: I think I've tracked down the bug: https://github.com/geek1011/dictutil...ment-629352018.

[Flaviocc]

Guys, I'm new in Kobo's world and lost with a few infos about dictionaries.

I have 2 files:

hyph_pt_br.zip --- with hyph_pt_br.dic inside (hyphenation dictionary)

pt_br.zip -- with pt_br.aff and pt_br.dic inside (ptbr dictionary)

can i use these dictionaries in kobo? And if so how?

I saw that in the patches there is a dictionary file, would it be there to insert the file?

Post here to avoid a new topic about dictionaries.

Thanks for help me.

[loyukfai]

Just upgraded to 4.21.15015

Unfortunately my custom translation dictionaries are still being replaced on each sync.

Do you think if it's possible edit the sqlite db, perhaps the size? So that the system won't "detect" the anomaly?

Cheers.

[geek1011]

Originally Posted by loyukfai

Just upgraded to 4.21.15015

Unfortunately my custom translation dictionaries are still being replaced on each sync.

Do you think if it's possible edit the sqlite db, perhaps the size? So that the system won't "detect" the anomaly?

Cheers.

Not anymore. The only option right now is the patch here: https://pgaskin.net/dictutil/dicthtml/install.html.

[loyukfai]

Originally Posted by geek1011

Not anymore. The only option right now is the patch here: https://pgaskin.net/dictutil/dicthtml/install.html.

Noted. Which option do I need to enable in /src?

Cheers.

[geek1011]

Originally Posted by loyukfai

Noted. Which option do I need to enable in /src?

Cheers.

Copy that into the yaml file for libnickel, then run kobopatch.

[ICDeadPpl]

Originally Posted by geek1011

Not anymore. The only option right now is the patch here: https://pgaskin.net/dictutil/dicthtml/install.html.

Maybe add to the page where the patch is supposed to be added?

[loyukfai]

Originally Posted by geek1011

Copy that into the yaml file for libnickel, then run kobopatch.

Thanks a lot! Done and working.

Wish Kobo will revert the change, so no need to do this manual action every time the firmware is updated...

Cheers