Suggestion: Spellcheck Enhancement (Numbers)

[Tex2002ans]

Currently, Sigil does not consider Numbers as "words".

This means that Spellcheck can't catch entire classes of errors, because many "words" don't display in the Spellcheck List (Tools > Spellcheck > Spellcheck).

I've attached an Example EPUB to show the issue.

The Problem

This includes the "Current Spellcheck List" vs. "Proposed Spellcheck List" in Spoilers.

Example 1: Centuries or Years:

In the 21st century, [...]
In the 1800’s, there was [...]

Spoiler:

In          In
the         the
st          21st
century     century
s           1800’s
there       there
was         was

Example 2: Pounds/Shillings/Pence/Money

The device cost £14 8s 2d.

Spoiler:

The         The
device      device
cost        cost
            14
s           8s
d           2d

Example 3: Hyphenated Years or Age:

In the 10-year period between [...]
The 10-year-old girl [...]

Spoiler:

In         In
the        the
year       10-year
period     period
between    between
The        The
year-old   10-year-old

Example 4: Weights/Measures

It weighs 100.5lbs.
The length is 100.5km and 2ft.

Spoiler:

It         It
weighs     weighs
lbs        100.5lbs
The        The
length     length
is         is
km         100.5km
and        and
ft         2ft

Example 5: Indexes/Footnotes

Dogs, 123n., 125, 130n.
See p. 123ff.

Spoiler:

Dogs      Dogs
n         123n
          125
n         130n
See       See
p         p
ff        123ff

Example 6: A very common typo (especially because of OCR):

In the 196os, the president was [...]
In l941, the samples were [...]
Good argument, h0wever, you are [...]

Spoiler:

os        196os
l         l941
h0wever   h0wever

It in Action

Calibre already includes numbers in their Spellcheck:



and it is extremely helpful.

Proposal

There is one downside to the Calibre-method though, because the Spellcheck List gets flooded with numbers. Especially when dealing with HTML tables full of data (or in Indexes):



To get around that issue:

• Ignore all "words" made completely of numbers + punctuation
  
  • Although I could see a usage for still keeping this (catching typos)
• Include the "Numbers as Words" as a checkbox/toggle.
  
  • Similar to "Show All Words"
    
  • Probably default to OFF.
    
  • This could allow the user to choose whether they want to display those full-numbers or not.

[Doitsu]

I also think that a Numbers as Words checkbox would be a good idea, in particular for OCRed text.

I looked into this some time ago and found out from KevinH that a toggle would need to change line #143 in /src/Misc/HTMLSpellCheck.cpp from:

    if (c.isLetter()) {

to

    if (c.isLetterOrNumber()) {

I.e., in terms of programming this shouldn't be too complicated to implement.

@KevinH, @DiapDealer:

Since this line appears to define what a "letter" is in terms of spell-checking, it should be possible to add curly right apostrophes (’ U+2019) to the list of "letters" via an additional if clause. This would fix another frequently reported spell check problem.
To keep it simple, the algorithm would only have to accept curly right apostrophes in the middle of words.

EDIT: I must have misremembered this.

[doubleshuffle]

I would also appreciate such a feature. I'm working with a scan right now where the OCR has produced tons of "l" vs. "1" mix-ups. They are very hard to detect with the spellcheck as it is now.

[Tex2002ans]

Originally Posted by doubleshuffle

I would also appreciate such a feature. I'm working with a scan right now where the OCR has produced tons of "l" vs. "1" mix-ups. They are very hard to detect with the spellcheck as it is now.

Currently, as a workaround, I use Calibre's Spellcheck (like pictured above).

Then I usually just type the numbers 0-9 in the search box one-by-one, and do a quick scan through the list to see if anything strange pops out.

And I have a few Regex that I use to try to minimize the impact:

Search: [lo]\d
Search: \d[lo]

That tries to catch things like "19l0" or "l910" or "It was 8.o5cm long".

I also try to use this:

Search: (Jan|Feb|Mar|Apr|Aug|Sept|Oct|Nov|Dec)\. [lo]
Search: (January|February|March|April|August|September|Oct ober|November|December) [lo]

to try to catch the odd dates: "Jan. i5, 2017" or "March i, 1910" or "August i982".

[doubleshuffle]

Thanks. I'll try that.

[KevinH]

Originally Posted by Doitsu

Since this line appears to define what a "letter" is in terms of spell-checking, it should be possible to add curly right apostrophes (’ U+2019) to the list of "letters" via an additional if clause. This would fix another frequently reported spell check problem.
To keep it simple, the algorithm would only have to accept curly right apostrophes in the middle of words.

As explained in that thread....

That is not a spellcheck bug, but a bug in the design of the German Hunspell dictionaries used.

[Doitsu]

Originally Posted by KevinH

As explained in that thread....

That is not a spellcheck bug, but a bug in the design of the German Hunspell dictionaries used.

Sorry. I must've misremembered this. I thought this problem also occurred with English spellcheck dictionaries.

[KevinH]

Understood.
For the record, spellchecking works with utf-8 encoded dictionaries that actually have words with apostrophes (contractions and the like) in that dictionary wordlist.

The problem with the German dictionary in question is twofold:

1. It was encoded is iso-8859-1 not utf-8 and the smart single quote/ apostrophe does not exist in that single byte encoding
2. The dictionary wordlist itself did not include any words with apostophes (single quotes) in the dictionary at all.

These are problems the dictionary owner should fix.

That said, I will see about adding support and a preference setting for numbers. But please note: not all dictionaries add mixed letter number words to their dictionary wordlist, and so perfectly valid things will then get marked as bad.

Grepping for all digits and examining them is probably a safer approach.

[Tex2002ans]

Originally Posted by Doitsu

Since this line appears to define what a "letter" is in terms of spell-checking, it should be possible to add curly right apostrophes (’ U+2019) to the list of "letters" via an additional if clause. This would fix another frequently reported spell check problem.
To keep it simple, the algorithm would only have to accept curly right apostrophes in the middle of words.

Actually, that reminds me of another Spellchecking bug that has been bothering me for a while.

I attached a Sample EPUB to show the problem.

Sample Code:

<p>The draft which TJ submitted to Hamilton is in Ford, VI, 7-69, with Hamilton’s notes and TJ’s comments on them.</p>

<p>The draft which TJ submitted to Hamilton is in Ford, VI, 7-69, with Hamilton’s notes and TJ's comments on them.</p>

In Code View, you can see both TJ’s (smart quote) + TJ's (dumb quote) is marked as a misspelled word.

If you Right Click on the first one + Ignore... nothing happens.

If you Right Click on the second one + Ignore... both lose their red squigglies.

This is pretty frustrating. I have a Keyboard Shortcut set to the "Ignore" function, and it becomes painstaking to work through larger books, because the "smart quote" words never get properly Ignored.

This is an issue on Windows (not too sure about other OSes).

Originally Posted by KevinH

That said, I will see about adding support and a preference setting for numbers.

[Tex2002ans]

I also thought of a few extra samples where numbers are "words".

Names of companies:

23andme tests your DNA.

Spoiler:

andme         23andme
tests         tests
your          your
DNA           DNA

Misc.:

Write this on A4 paper.
You are in Room B9.
This is a B-17 Bomber.

Spoiler:

A             A4
B             B9
B             B-17

It showing up in the Spellcheck List makes it very easy to jump to the location in the EPUB with a doubleclick.

Currently, the "A" in "A4" would be impossible to spot easily (there could be thousands of A in the book). And B9 + B-17 wouldn't stand apart.

Originally Posted by doubleshuffle

Thanks. I'll try that.

Oh yeah, and another common OCR error you might run across:

Search: 8c
Replace: &
(or in the case of HTML) Replace: &amp;

8c is also something that sticks out easily in Calibre's Spellcheck List.

[KevinH]

I just tried this on Mac OS X with stock Sigil (most recent) and all of the TJ's and smart versions are marked as incorrectly spelled in CodeView on first start-up.

I then modified your test case to include a duplicate of each of your 2 lines so that I could see if later versions are properly ignored or not (see below).

<body>
  <p>The draft which TJ submitted to Hamilton is in Ford, VI, 7-69, with Hamilton’s notes and TJ’s comments on them.</p>

  <p>The draft which TJ submitted to Hamilton is in Ford, VI, 7-69, with Hamilton’s notes and TJ's comments on them.</p>

  <p>The draft which TJ submitted v2 to Hamilton is in Ford, VI, 7-69, with Hamilton’s notes and TJ’s comments on them.</p>

  <p>The draft which TJ submitted v2 to Hamilton is in Ford, VI, 7-69, with Hamilton’s notes and TJ's comments on them.</p>
</body>

I then right clicked on the red squiggly line of the first smart version and selected "Ignore".

And correctly all later versions (both smart and dumb) are properly now not marked as wrong.

So I can not recreate your issue at all.

So what exact version of Sigil are you using? What version of Qt is it using? Have you installed your own dictionaries and if so which?

Originally Posted by Tex2002ans

Actually, that reminds me of another Spellchecking bug that has been bothering me for a while.

I attached a Sample EPUB to show the problem.

Sample Code:



In Code View, you can see both TJ’s (smart quote) + TJ's (dumb quote) is marked as a misspelled word.

If you Right Click on the first one + Ignore... nothing happens.

If you Right Click on the second one + Ignore... both lose their red squigglies.

This is pretty frustrating. I have a Keyboard Shortcut set to the "Ignore" function, and it becomes painstaking to work through larger books, because the "smart quote" words never get properly Ignored.

This is an issue on Windows (not too sure about other OSes).

[Doitsu]

@KevinH:

This is most likely a Windows issue. On my Linux machine, right-clicking TJ’s in the first line and selecting Ignore removed all red squiggly lines.
(Right-clicking TJ's in the second line and selecting Ignore had the same effect.)

However, when I repeated the test on my Windows machine, right-clicking TJ’s in the first line and selecting Ignore didn't remove the red squiggly lines from any line.
When I right-clicked TJ's in the second/fourth line and selected Ignore, all red squiggly lines were gone. (I used Sigil 0.9.8 and the stock en_US and en_GB dictionaries for all tests.)

[KevinH]

Since it is not Sigil versions specific and not Qt specific, I wonder if this is Locale dependent? What Locale is set for Windows? What happens if you try other Locales? What is your default Windows encoding as well?

I guess it could be a Windows Qt specific bug? Perhaps they are confusing unicode smart quotes with Windows specific encoding smart quotes? Something funny is going on.

[Tex2002ans]

Originally Posted by KevinH

So I can not recreate your issue at all.

So what exact version of Sigil are you using? What version of Qt is it using? Have you installed your own dictionaries and if so which?

(This also affected my Windows 7 machine for a looong time. This has been in Sigil for a while now.)

Windows 10, 64-bit.
Sigil 0.9.8
Qt 5.6.2

Never touched a dictionary file in my life. It's just whatever comes in default Sigil.

(This is just the default Sigil install, right from the site, no funny business.)

Originally Posted by KevinH

Since it is not Sigil versions specific and not Qt specific, I wonder if this is Locale dependent? What Locale is set for Windows? What happens if you try other Locales? What is your default Windows encoding as well?

America! :P

Locale: English (United States)

Where can I check the "default Windows encoding"?

I'm assuming whatever the defaults are in a Windows 7/10 install. Never messed with those settings.

[DiapDealer]

My test on Windows Vista mirrors's Doitsu's results: clicking ignore on the one with the smart-quote had no effect ... clicking ignore on the one with the straight-quote removed the red misspelled line for all four occurrences (smart and dumb).