Sigil Dictionary Update

[KevinH]

Hi All,

Please let me know if there are any other truly obsolete words in these spellchecker dictionaries that really should be moved to size 80 and not appear in size 60 or 70. I can make those changes in my local copy of scowl before building the final versions of these dictionaries later this week.

If the differences get significant enough, Sigil can just create its own fork of scowl and and maintain it for our own internal use.

The same goes for any of the dictionaries we include with Sigil on Windows and macOS. If there are better versions of any of these other language dictionaries, we would be happy to update what we have in our repo (assuming the license is at all compatible). But please be critical in your evaluations.

Thanks,

KevinH

[Ashjuk]

Originally Posted by KevinH

A spellchecker dictionary is very different from a regular dictionary. Its role is to help catch common spelling errors, not to define a language or a word. Therefore keeping obsolete or very rarely used words in a spellchecker dictionary is just not appropriate given they help hide spelling errors on more commonly used words.

I will remove both Scotchman, and Scotchwoman and their variants but leave scotchman in the final release. If people have a historical text that uses those words that they do not want to update to their modern equivalents, they can easily ignore those words or simply add them to their User dictionary.

This is the same reason a spellchecker dictionary should not be based on scowl size 80 or larger (and many say 70 or larger).

Unfortunately, vetting the scowl word lists really requires a team of dedicated people not just one or two.

Thanks,

KevinH

Probably the best solution. Anyone misspelling Scotsman etc. should be using a capitalised version.

I will keep an eye out for anything else that might cause an issue.

[KevinH]

Hi All,

I extracted a few common words missing from the google .dic_delta files and added them in, moved Scotchman and etc to level 80, properly add the no suggest flags, added in the checked and new words contributed by Ashjuk and added the proper README files from scowl.

See the attached. Unless people complain, I will be using the size 70 to update the ones in the Sigil repo tree before the next release.

Thanks to all who commented and helped.

[Ashjuk]

One thing I have noticed since using the new dictionary is that sometimes the suggested replacements for misspellings is a little off.

I came across self-defense in a book today. I thought that right clicking would bring up self-defence as an option, but I was surprised to see that it is not even listed.



I also noticed that when using Spellcheck the suggested replacements is a bit off. I was offered just about everything other than ploughed when I went to replace plowed.

[KevinH]

Suggestions are really based on edit-distance which is roughly measured as the number of single characters changes.

So to go from:

plowed

to

ploughed

is actually quite far in terms characters added, swapped, and removed, which is why phonetic suggestions such as used by aspell help.

As for self-defense vs self-defence, that is only a 1 character swap so as long as
self-defence is in the en_GB dictionary then it should have made the list. I will check if "self-defence" is in the dictionary.

[KevinH]

We can handle plowed vs ploughed via a .aff replacement table entry.

ow -> ough

We can add that.

Update:

Adding the ow -> ough to the replacement table in the en_GB.aff did the trick.
But neither self-defense nor self-defence exists in any of the word lists from scowl.

I think scowl appears to assume all words break on "-" but that is of course nonsense. That behaviour is up to the dictionary itself as not every valid combination of two words with a "-" in between is a valid word.

So I added self-defence to my en_GB dictionary and added the replacement table entry and now get the following:

See attached images.


Please let me know if you see any other irregularities.

[KevinH]

@Ashjuk
Here is an updated en_GB to continue your testing with.

[Ashjuk]

Originally Posted by KevinH

@Ashjuk
Here is an updated en_GB to continue your testing with.

Thanks Kevin, I will test that one out.

[KevinH]

Because scowl seems to not have even one hyphenated word in its word list I have found a list of the most common hyphenated words and will add them before the final release:

Here is the most-common list. Additions or corrections welcome.

able-bodied
anti-theft
brother-in-law
brother-in-law's
brothers-in-law
brothers-in-law's
call-up
check-in
clean-cut
co-worker
co-workers
co-worker's
corn-fed
daughter-in-law
daughter-in-law's
daughters-in-law
daughters-in-law's
de-emphasize
de-emphasized
double-cross
double-crossing
double-crossed
double-park
double-parking
double-parked
empty-handed
ex-husband
ex-husband's
ex-husbands
face-saving
family-run
father-in-law
father-in-law's
fathers-in-law
fathers-in-law's
follow-up
four-letter-word
front-runner
front-runner'ss
front-runners
front-running
full-time
get-together
get-togethers
good-looking
habit-forming
half-witted
high-spirits
high-spirited
high-tech
ill-timed
in-depth
know-it-all
large-scale
left-handed
life-size
life-sized
long-term
low-grade
low-key
merry-go-round
middle-aged
mother-in-law
mother-in-law's
mothers-in-law
mothers-in-law's
near-sighted
non-starter
not-for-profit
off-peak
off-site
old-fashioned
on-campus
one-half
one-sided
over-the-counter
part-time
passer-by
price-fixing
quick-witted
round-trip
run-in
runner-up
self-service
self-serving
short-change
short-changed
shrink-wrap
shrink-wrapped
single-minded
state-of-the-art
strong-arm
three-dimensional
tie-break
tie-breaker
tip-off
toss-up
two-fold
two-thirds
u-turn
u-turns
ultra-violet
up-to-date
walk-on
warm-up
well-being
well-known
word-of-mouth
worn-out
x-ray

[KevinH]

Alternatively, I could remove the - from the definition of WORDCHARS in the .aff and then every hyphenated word that was comprised of other correctly spelled words would automatically be deemed correct. I think this is how scowl expects things to be but not one I particularly think is appropriate.

[BetterRed]

IMO brother-in-laws should probably be brothers-in-law. Adding an 's' to the end of brother-in-law pluralises 'law'.

singular possessive - brother-in-law's
plural possessive - brothers-in-law's

BR

[KevinH]

Thanks ... I found a second list of the hyphenated words and it had over 47000 entrees.

I think the only solution is to remove the "-" from the WORDCHARS and accept anything that is okay when split into separate words at the hyphen. It seems scowl was based off this assumption and I really do not want to have to add and maintain 40+ thousand hyphenated words. Our old dictionary used this approach as well.

It is always something!

And I learned something new. Hunspell is smarter than MySpell and will actually try splitting hyphenated words and checking each one automatically whether there is a "-" in Wordchars or not.

So that means we really only need to add hyphenated words that are not already covered by that rule which seems to cover a lot of words, just not self defense or self-defence.

So it looks like I can take my list of 47000 hyphenated words and spellcheck them using hunspell to see how many of them actually need to be special cased other than self-defense/self-defence.

[roger64]

Hi

Provide a dictionary for French speaking users ?

As far as I can remember, I've been using the Grammalecte Hunspell French dictionary both with Sigil and the Calibre editor.

Grammalecte tools are open source and have been perfected over the years -and still are- by an extended community of enthusiast users. Count me among them.

Its dictionary has been extensively tested. We are currently at version 7.

Its grammar checking tool has been already made available for Sigil users thanks to a Doitsu plugin which is automatically updated at each new version of the tool.

As you can see on the screenshot below, following the recommendation of his author, I use by default the "classic" version of this dictionary but keeps loading the other ones, if need be.

Not every French speaking user is aware of it. I think it would be useful if Sigil could also recommend or even better select this Grammalecte dictionary by default. You'll find the precise page to download the latest version here:

https://grammalecte.net/download.php?prj=fr

Click on the green star "Dictionnaires Hunspell 7.0

[KevinH]

We can only include it to replace our current fr dictionary if its license is compatible. We do not have a way to recommend specific dictionaries to people. Perhaps a sticky thread here of some sort with recommended dictionaries might be a way to deal with this?

[roger64]

Originally Posted by KevinH

We can only include it to replace our current fr dictionary if its license is compatible.

This would be the best solution.

Grammalecte is published under GNU GPL v3.