Spellcheck Sigil 1.91

[Mbear]

High has anyone noticed that spellcheck in Sigil picks up words like the name of a person such as Clayton at the end of a sentence and includes the full stop and shows both Clayton and Clayton., as being incorrect. This behavior appears to be a recent event.

[KevinH]

Yes, the built-in dictionary has enabled proper spelling of abbreviations and things like "etc.". Before, the dictionary improperly accepted "etc" as being correct and it spell checked "F.B.I." as 3 separate words "F", "B", and "I". These short incorrect "root" words are then used to generate multiple bad suggestions.

The new dictionary allows the the ending period to be included with unknown words in case they are abbreviations. So words like "etc." and "F.B.I." can be checked properly. It then checks the word with and without the ending period and if that word is not in the dictionary, it reports it as misspelled.

The suggestions generated will include words variations with and without the ending period so the user can select what that ending period is meant to do.

If you want to revert to the earlier behaviour, install an older hunspell dictionary into your Sigil Preferences and it will disable that new feature as the dictionary itself has to be designed to support proper spellchecking of abbreviations.

[elibrarian]

Originally Posted by KevinH

Yes, the built-in dictionary has enabled proper spelling of abbreviations and things like "etc.". Before, the dictionary improperly accepted "etc" as being correct and it spell checked "F.B.I." as 3 separate words "F", "B", and "I". These short incorrect "root" words are then used to generate multiple bad suggestions.

The new dictionary allows the the ending period to be included with unknown words in case they are abbreviations. So words like "etc." and "F.B.I." can be checked properly. It then checks the word with and without the ending period and if that word is not in the dictionary, it reports it as misspelled.

The suggestions generated will include words variations with and without the ending period so the user can select what that ending period is meant to do.

If you want to revert to the earlier behaviour, install an older hunspell dictionary into your Sigil Preferences and it will disable that new feature as the dictionary itself has to be designed to support proper spellchecking of abbreviations.

I've tried to get to terms with this behaviour, but still after four or five months find it extremely irritating, and I don't want to use old dictionaries (maybe ok now, but in ten years time …). So I found out that in the danish hunspell .aff-file of 2022 there is a line:

WORDCHARS -.

Remove the period (and restart Sigil, if it is running), and the spellchecker will revert to (what I call) normal, that is no double entries of "misspelled" words with and without a period.

(And of course, if the dog in reality lies buried somewhere else in the spellchecker, it would be nice to know the correct way of toggling it.)

Regards,

Kim

[KevinH]

Sigil creates its own dictionary to enable this feature. No hunspell standard dictionary does this that I know about. Yes simply removing the period from WORDCHARS in any aff will change the break iterator algorithm resulting in the return of the old behaviour with etc and F B I being considered correct.

[Tex2002ans]

Originally Posted by Mbear

High has anyone noticed that spellcheck in Sigil picks up words like the name of a person such as Clayton at the end of a sentence and includes the full stop and shows both Clayton and Clayton., as being incorrect. This behavior appears to be a recent event.

This change occurred in March 2022, Sigil 1.9.1.

The changelog said:

• Extend spellcheck suggestions to properly handle end of sentence vs abbreviations

Originally Posted by elibrarian

So I found out that in the danish hunspell .aff-file of 2022 there is a line starting with

REP $_ ._

and if I comment that out (putting a # before REP $_ ._ ), the spell checker in Sigil reverts to the old behavior.

[...]

I can't find that line in the english hunspell .aff files, and haven't checked other languages. But maybe this can help someone, somewhere

Fantastic. Thank you for that info.

Originally Posted by elibrarian

(And of course, if the dog in reality lies buried somewhere else in the spellchecker, it would be nice to know the correct way of toggling it.)

Yeah, I spoke about this in private to KevinH when it was implemented, and STRONGLY recommended against it.

The amount of acronym-periods vs. sentence-ending-periods are astronomical.

And to heavily weigh Spellcheck Lists:

• Correct acroynms vs. Wrong sentence-enders (Sigil 1.9.1+)
  
  • "F.B.I." vs. "Clayton."
    
• vs. "Wrong" acroynms vs. Correct sentence-enders (<=Sigil 1.9.0)
  
  • "F.B.I" vs. "Clayton"
    

It also completely distorts the sortability/skimmability of lists + accuracy of word counts + makes it more annoying to "Ignore" all cases of a misspelled word.

There just isn't any comparison in my mind.

The amount of correct acronyms would be heavily outweighed by "wrong" spelling (red squigglies) on every single book's usage of sentence-ending periods.

Originally Posted by KevinH

If you want to revert to the earlier behaviour, install an older hunspell dictionary into your Sigil Preferences and it will disable that new feature as the dictionary itself has to be designed to support proper spellchecking of abbreviations.

This is A solution, but I still don't think it's a user-friendly one.

I'll continue being grumpy about this change... but great work on the rest of Sigil's enhancements though! Still my favorite EPUB editor ever!

[DiapDealer]

Originally Posted by Tex2002ans

I'll continue being grumpy about this change...

It's not like there's any danger of running out of misplaced grumpiness anytime soon.

But I have to ask: why waste time being grumpy when you can substitute your own hunspell library (or libraries) that work exactly like you want? There's no point in suffering when the workaround is easy enough is there?

[KevinH]

Plus this occurs only with words that are *not* in the dictionary at the end of a sentence. An example here might be a proper first name. Words in the dictionary that end a sentence are properly checked. So comparing end of sentence words against abbreviations and acronyms is not the correct comparison.

And as DiapDealer said, just install your own hunspell dictionary that is not built to handle abbreviations if you so desire.

[Tex2002ans]

While I was gathering this info, I believe I found a bug in Sigil 1.9.10:

1. Paste this code into a book:

<p>This is an example of a <i>sentence-ender</i>. That continues for another sentence.</p>

2. Press Tools > Spellcheck > Spellcheck.

3. Search: .

The period after "sentence-ender" will show up as a "word" by itself.

(Because of the closing italic + .)

- - -

Enhancement Request: Think we could get the Spellcheck List "Count" column aligned right?

Similar to my 2021 Reports columns alignment request!

For easier readability/comparability.

- - -

Enhancement Request #2: Think we could get the "Language" + "Mispelled?" columns flipped?

Before:

Word        | Count | Language               | Misspelled?
____________|_______|________________________|_____________
example     | 1     | English (United States)| No
examples    | 2     | English (United States)| No
exampled    | 3     | English (United States)| No
exampleness | 4     | English (United States)| Yes
español     | 5     | Spanish                | No

After:

Word        | Count | Misspelled? | Language
____________|_______|_____________|_____________
example     | 1     | No          | English (United States)
examples    | 2     | No          | English (United States)
exampled    | 3     | No          | English (United States)
exampleness | 4     | Yes         | English (United States)
español     | 5     | No          | Spanish

I think that would really help with readability.

Plus, if you didn't care about Language, you could easily resize the window slightly to chop it off. :P

(And some of those Languages are REALLY verbose!)

- - -

Originally Posted by DiapDealer

But I have to ask: why waste time being grumpy when you can substitute your own hunspell library (or libraries) that work exactly like you want? There's no point in suffering when the workaround is easy enough is there?

Yes, yes, I know.

But for a common user (like Mbear or GreggBell) to know where to dig and "fix this" and substitute with an old hunspell, it's just madness.

Put a foot in the normal human's shoes for a second!

- - - -

Introductory Note: I believe making this acronym/sentence-ender optional would help, similar to:

• Edit > Preferences > Spellcheck Dictionaries
• "Check Numbers" checkbox

Maybe that UI can be adjusted slightly, and a new checkbox introduced:

• Check Periods
  
  • Off = pre-Sigil 1.9.1
    
  • On = Sigil 1.9.1
    

(Or some much better name. )

(And, personally, I'd argue for it to be OFF by default. See reasoning below.)

(Advanced users can then turn it ON if needed—just like "Check Numbers".)

- - - -

The Big Picture on Closing-Periods-as-Words

The largest problem I have is:

• The # of hits you have to look through explodes.
  
  • Nearly doubling.
• Word Counts are thrown off.

The entire purpose—and extreme power—of Spellcheck Lists is to be able to compare/sort + get accurate counts.

This allows you to quickly see, at-a-glance, problems which would have otherwise been hidden or very hard to spot:

peeked    |   5
peaked    |   1
Rothbard  | 100
Rothbird  |   2
Rotbard   |   1
Malone    |  20
Molone    |   2
Mises     |  50
Misses    |   3

What Sigil 1.9.10 does, is something like this:

peeked    |   3
peeked.   |   2
peaked    |   1
Rothbard  |  60
Rothbard. |  40
Rothbird  |   1
Rothbird. |   1
Rotbard   |   1
Malone    |  19
Malone.   |   1
Molone    |   1
Molone.   |   1
Mises     |  40
Mises.    |  10
Misses.   |   3

We've gone from 9 -> 15 hits, and potential typos get buried in the morass of:

• word
• + same exact word.
• new word
• + same exact new word.

- - -

Lost in the Clutter

For example, before, these show up right next to each other:

Spoiler:

Rothbard
Rothbird <--- This looks strange
Rotbard  <--- This is 1 character short, definitely stands out.

peeked
peaked   <--- 1 letter diff.

Malone
Molone   <--- 1 letter diff.

Mises
Misses   <--- 1 character extra, definitely stands out.

Now, you have a "visual clutter" full of:

Spoiler:

3 -> 5
Rothbard
Rothbard.
Rothbird  <--- Doesn't look like a typo, at a glance
Rothbird. <--- because it "shows up twice".
Rotbard


2 -> 3
peeked
peeked.
peaked  <--- Hard to spot because word above isn't
             same exact size-but-one-letter-diff.

2 -> 4
Malone
Malone.
Molone  <--- I probably wouldn't check this name because
Molone. <--- it shows up twice + looks like any other.


2 -> 3
Mises
Mises.  <--- This type's especially hard to spot
Misses       because the '.' is EVERYWHERE, so it
             "merges" in with the extra 's' in the word below

This at-a-glanceness gets worse when you SORT by Count:

Spoiler:

Rothbard   100
Mises       50
Malone      20
peeked       5
Misses       3
Molone       2
Rothbird     2
peaked       1
Rotbard      1

Rothbard   60
Mises      40
Rothbard.  40
Malone     19
Mises.     10
Misses.     3
peeked      3
peeked.     2
Malone.     1
Molone      1
Molone.     1
peaked      1
Rotbard     1
Rothbird    1
Rothbird.   1

How many times did "Malone" show up in this book? 20 times.

But not according to the period-sort! 19+1.

• Before, typos showed up in rows right near each other.
  
  • Now, typos are "hidden" with periods.
    
  • What would have been 1 correct + 1 typo becomes 1/2 correct + 1/2 typos.
    
• Before, counts gave you real word counts.
  
  • After, they give you partial word counts.
    
  • And they'll appear far apart from each other + have to manually be added.
    

This makes comparison extremely hard.

- - -

Side Note: One of most common typos is seeing a spelling 10+ times, and a similar spelling 1 time, think:

color   | 10
colour  |  1

Now, it shows up as:

color   |  9
color.  |  1
colour  |  1

You might not see "colour", because it's not "10 vs. 1" but "1 vs. 1"!

- - -

See some comparison images.

Before vs. After:




Much fewer words per screen, and near-words (or typos) get lost under all the "duplicate periods".

Plus, your eyes are always "stutter-stepping", because of the:

• 1 character smaller
• + 1 character bigger
• 1 character smaller
• + 1 character bigger
  • Oh, great, another period...

When 99% of these extra "words" are duplicate periods... your brain turns off.

- - -

That's the major problems, as I see it, but I've got many other intermediate/smaller ones too.

Here's a few I picked out of the last book I worked on:

Problem #1: Indexes/PageNumbers (especially roman numerals)

Similar to Spellcheck Lists in Calibre getting flooded with numbers, this index/roman numeral issue also kicks it up to 1000!

You get hundreds and hundreds of extra:

i.
ii.
ix.
I.
II.
III.

clogging almost every single search—again, drowning out many of the otherwise at-a-glance issues.

Problem #2: URLs Lost

One trick I love/d to use is a period to find yet-to-be-linked URLs in a book:



URLs are completely lost in Spellcheck Lists now.

Not just by (partial) acronyms but by every single sentence now too!

Problem #3: URL "sentence-enders"

I've seen lots of nearly-doubling:

• example.com
• example.com.
• example.gov
• example.gov.
• example.pdf
• example.pdf.
• exaample.pdf
  
  • (Again, this is a very common area for typos to hide.)

When you work on citations, this becomes a huge problem!

- - -

Do I need to continue? I have lots more examples!

The current way of Sigil's new default spellchecking is unlike any other program there is—and not in the good way!

- - -

Anyway, like I said, a nice, easy-to-use checkbox would be a nice addition. (OFF by default!)

Then we could say:

• "Hey, Mbear! Do you want this weird, advanced way of spellchecking?"
• "Go into Preferences > Spellcheck Dictionaries and enable those 2 boxes!"

Those users who know what they are doing + have specific cases for it can enable it.

But for the love of all that is holy, put advanced stuff as options for the advanced users!

- - -

PS. I still love you, KevinH and Diap, but sometimes I want to just hug your little necks with two hands!

[DiapDealer]

Yes, yes, I know.

But for a common user (like Mbear or GreggBell) to know where to dig and "fix this" and substitute with an old hunspell, it's just madness.

Put a foot in the normal human's shoes for a second

Ah... I feared it might be a "think of the children" thing.

But "madness"?? That's a tad extreme. The ability to add custom hunspell dictionaries has long been a feature of Sigil. Was it madness to offer that ability? Don't get hung up on needing to get an "old" hunspell dictionary. "Old" was in the sense that any version of Sigil previous to 1.9.1 would have one. ANY non-Sigil-1.9.1 hunspell dictionary will suffice. It doesn't need to be "old".

[Tex2002ans]

The normal children—ages 3+!

[CubGeek]

[Turtle91]

Way to throw mbear and GregBell under the bus!! Lol

I actually played around with this spellcheck stuff… and the periods after a duplicate word was a slight annoyance, but not that big a deal. Once you add the root word to a dictionary you can refresh the spelling list… it finds the word(s) in the dictionary and doesn’t display them in the misspelled list anymore. The refresh is very fast.

The slight annoyance was from having to move the mouse back and forth between selecting the word and then clicking a button - when you have to skip every word with a period. Otherwise you would have both words and period words in the dictionary. Any way to make the spellcheck buttons return focus to the wordlist - preferably the word just below the previous location?? That will allow people to use the arrow keys to skip over any words (dotted or otherwise) while not moving the mouse.

[KevinH]

Just install any Hunspell en dictionary and the previous behaviour you want is restored. So no checkbox preferences or ui changes will be made. No editing of aff is needed. A dictionary that supports abbreviations and reduces spurious suggestions needs to be designed to do so. It uses different word lists. So using just a checkbox preference is not possible. But adding a hunspell dictionary is easily done by users.

Trying to misuse spellcheck to check for bad urls when a plugin exists to do that seems a bit over the top.

I will look into reordering the columns and right adjusting the counts, just NOT for the next release as that would entail large changes in the User Guide spellcheck related images when we are just about to make a new release.

Yes, adding the word to the dictionary will remove it sentence ending variant when a refresh is done.

So just install your own hunspell en dictionary into your Sigil Preferences and you will never have to live with it. I work on many math heavy papers typically loaded with abbreviations and acronyms but with very few or next to none proper first names. I like the hunspell dictionary I built to handle that aspect more correctly. If you do not like that behaviour simply install any english hunspell dictionary (newer or older) just once to get the behaviour you want.

And if you want to make accurate counts, I recommend using the new Saved Search Group Counts Report feature and not trying to use SpellCheck for that. It was added for just that purpose.

[elibrarian]

Originally Posted by DiapDealer

ANY non-Sigil-1.9.1 hunspell dictionary will suffice. It doesn't need to be "old".

That is not correct, since the 2022 Danish not-Sigil hunspell dictionary has this "feature".

As for how un-annoying it is, so - yes, if the text you are checking i in the same language as the spell checking dictionary, without many foreign word and places, so … But translated works, playing out in far & foreign places …. I have just spell checked Conan Doyle's Brigadier Gerard stories, and since the good brigadier gets around in the world, there's a lot of "misspelled" words at the end of lines - oh, SOO much fun. And I didn't find one - not one! – misspelled abbreviation, that would have justified the thing.

Regards,

Kim

[KevinH]

ps. Since it is only a one line change that will not completely invalidate the user guide spellcheck images, I have aligned the count field right (numerically). Any change in column order will have to come in a future release just not our upcoming one.