How to change fuzzy search threshold (6.8+)

[datyoma]

In 6.8 firmware dictionary lookup algorithm was completely rewritten. These days, it starts with fuzzy search, which can be frustratingly overzealous (https://www.mobileread.com/forums/sh...&postcount=482); and if it finds nothing, various morphological engines are tried in turn, including Hunspell.

Fuzzy search discards candidates based on a normalized edit distance threshold, which is hardcoded to 0.4. So if you want it to be less aggressive, this threshold needs to be lower.

Turns out, changing it is quite simple as long as you have root/ssh access (follow the instructions in https://www.mobileread.com/forums/sh...d.php?t=325185).

SSH into the device and do the following as root - this should change the threshold to 0.1 on 6.10; for 6.8, use 1f instead of 0f:

$ mkdir -p /mnt/ext1/patches
$ xxd -p /ebrmain/cramfs/lib/libdictionary.so | tr -d '\n' | \
    sed 's/d90f43e3/b90f43e3/' | \
    xxd -r -p > /mnt/ext1/patches/libdictionary.so
$ mount -o remount,rw /ebrmain/
$ rm /ebrmain/lib/libdictionary.so
$ ln -s /mnt/ext1/patches/libdictionary.so /ebrmain/lib/libdictionary.so

What's going on here, you might ask? Let's look at these bytes with objdump:

$ objdump -d libdictionary.so | grep -C8 '0x3fd9' | c++filt 
   4f30c:       e92d47f0        push    {r4, r5, r6, r7, r8, r9, sl, lr}
   4f310:       e24dd0c0        sub     sp, sp, #192    @ 0xc0
   4f314:       e1a06001        mov     r6, r1
   4f318:       e309199a        movw    r1, #39322      @ 0x999a
   4f31c:       e1a04000        mov     r4, r0
   4f320:       e3090999        movw    r0, #39321      @ 0x9999
   4f324:       e28d8068        add     r8, sp, #104    @ 0x68
   4f328:       e3491999        movt    r1, #39321      @ 0x9999
 ! 4f32c:       e3430fd9        movt    r0, #16345      @ 0x3fd9
   4f330:       e59d50e0        ldr     r5, [sp, #224]  @ 0xe0
   4f334:       e58d1000        str     r1, [sp]
   4f338:       e1a01006        mov     r1, r6
   4f33c:       e98d0021        stmib   sp, {r0, r5}
   4f340:       e1a00008        mov     r0, r8
   4f344:       e1a09003        mov     r9, r3
   4f348:       e1a07002        mov     r7, r2
   4f34c:       ebff5b1c        bl      25fc4 <pocketbook::dictionary::Dictionary::levenshteinLookup(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, double, std::atomic<bool> const*) const@plt>

3fd9 9999 9999 999a (stored in registers r0 and r1) is the IEEE 754 representation of 0.4 - you can paste it here for better understanding: https://ieee754-converter.utils.com/

The d9 byte includes 4 bits of the exponent and 4 bits of the mantissa, so tweaking this byte alone covers the range from 0.00003 (00) to 1.975 (FF), which is more than enough and avoids the need to understand the exact encoding of ARM instructions. In the example above, I changed it to b9, which corresponds to 0.1, but you can try out different values.

The rest of the script replaces the symlink (originally it points to ../cramfs/lib/libdictionary.so). If anything goes wrong, just change the symlink back - the version in cramfs remains read-only.
___________________________________

My impression is that fuzzy search tries to compensate for morphological engine deficiencies. Hunspell dictionaries that can be downloaded from PB servers are the usual spell-checking ones that you can find in LibreOffice - they are not optimized for stemming, for which you often need to explicitly map conjugations to base forms: Hunspell index.dic plays the same role as StarDict .syn file, so to say.

Therefore I invested quite a bit of time in creating custom Hunspell dictionaries for German and Spanish - hopefully someone finds them useful: https://codeberg.org/datyoma/hunspell-stemming-dicts (copy to /mnt/ext1/system/morphology)