Spellcheck and apostrophes

[AnselmD]

Originally Posted by varlog

There is this piece of code in Sigil:

QString Utility::getSpellingSafeText(const QString &raw_text)
{
    // There is currently a problem with Hunspell if we attempt to pass
    // words with smart apostrophes from the CodeView encoding.
    // There are likely better ways to solve this, but this one does
    // get the job done until someone can implement something better.
    QString text(raw_text);
    return text.replace(QString::fromUtf8("\u2019"), "'");
}

Don't know if it's related, it is used three times in SpellCheck.cpp. Just info.

Maybe this is done (and should be done) by the .aff fille?

if i remove the line
ICONV ’ '
from de_DE_OLDSPELL.aff
it works.
i can leave the following line untouched.
OCONV ' ’

[Doitsu]

Originally Posted by AnselmD

Where can i get working version of:
German (de-DE-1901) old spelling dictionaries-2016.04.03 | Apache OpenOffice Extensions
http://extensions.openoffice.org/de/...aries-20160403

@AnselmD: These dictionary files were already converted to utf-8 by the maintainers. Since you've already figured out how solve the problem by editing the .aff file, there's nothing for the developers to do.

@KevinH: There was a spelling reform in Germany in 1996 and many word processors and text editors offer spelling support for both "unreformed German" and "reformed German." Since many German MR contributors use Sigil to prepare German Public Domain MR books written according to the pre-1996 spelling rules, it'd be nice, if you could officially add the GPL licensed, utf-8 encoded de_DE_OLDSPELL.dic/aff spellcheck dictionaries from dict-de_de-1901_oldspell_2016-04-03.oxt in one of the next Sigil builds.

The OpenOffice developers have also released a new version of the post 1996 German spelling dictionaries in 01/2017.
Could you please extract de_DE_frami.dic/aff files from dict-de_de-frami_2017-01-12.oxt, convert both files to utf-8, change the SET ISO8859-1 line to SET UTF-8 and include these dictionary files (instead of the de_De dictionaries) in one of the next Sigil builds?
(As I've already mentioned, the OLDSPELL dictionaries for "unreformed German" have already been converted to utf-8 by the maintainers. I.e., all you have to is add them to the build scripts.)

[KevinH]

That line should only be removed if you decrement the count in the iconv header line. Also, the actual text encoding used in the .aff file matters. It is has to match that specified in the line with SET.

The byte value for the smart single quotes depends on encoding and must be correct for the SET line encoding.

[KevinH]

Originally Posted by Doitsu

@KevinH: There was a spelling reform in Germany in 1996 and many word processors and text editors offer spelling support for both "unreformed German" and "reformed German." Since many German MR contributors use Sigil to prepare German Public Domain MR books written according to the pre-1996 spelling rules, it'd be nice, if you could officially add the GPL licensed, utf-8 encoded de_DE_OLDSPELL.dic/aff spellcheck dictionaries from dict-de_de-1901_oldspell_2016-04-03.oxt in one of the next Sigil builds.

Users can simply install this dictionary on their own. I do not want to add the extra meg or so to include it on every download when it is of use only for older works and does not represent current spelling in German. Users can install their own dictionaries quite easily. The .oxt is pretty much just a zip file.

The OpenOffice developers have also released a new version of the post 1996 German spelling dictionaries in 01/2017.
Could you please extract de_DE_frami.dic/aff files from dict-de_de-frami_2017-01-12.oxt, convert both files to utf-8, change the SET ISO8859-1 line to SET UTF-8 and include these dictionary files (instead of the de_De dictionaries) in one of the next Sigil builds?

If the dictionary license is compatible with Sigil then yes we can do this.

KevinH

[Doitsu]

Originally Posted by KevinH

That line should only be removed if you decrement the count in the iconv header line. Also, the actual text encoding used in the .aff file matters. It is has to match that specified in the line with SET.

The byte value for the smart single quotes depends on encoding and must be correct for the SET line encoding.

Presumably, neither I nor the OP is familiar with the exact syntax of the AFF/DIC files and are therefore ill-equipped to provide any useful feedback to you.

Originally Posted by KevinH

If the dictionary license is compatible with Sigil then yes we can do this.

They're GPL2/3 licensed.

[KevinH]

The hunspell man pages and docs describe what can be used in an aff file. It was hunspell that added the iconv and oconv features. Before the chararacter conversions on input is an iconv number that tells how many iconv patterns follow it. If you remove one of the iconv pattern lines, you need to decrement that line count. Also if you edit the aff in the wrong encoding, you can mess it up.

Here is a good hunspell man page I found online that describes the contents of the aff file and its meaning:

http://manpages.ubuntu.com/manpages/...unspell.4.html

I will take a look at the dictionary and see why removing that line seems to help.

Thanks,

KevinH

[AnselmD]

Originally Posted by KevinH

The hunspell man pages and docs describe what can be used in an aff file. It was hunspell that added the iconv and oconv features. Before the chararacter conversions on input is an iconv number that tells how many iconv patterns follow it. If you remove one of the iconv pattern lines, you need to decrement that line count. Also if you edit the aff in the wrong encoding, you can mess it up.

Here is a good hunspell man page I found online that describes the contents of the aff file and its meaning:

http://manpages.ubuntu.com/manpages/...unspell.4.html

I will take a look at the dictionary and see why removing that line seems to help.

Thanks,

KevinH

Hi Kevin,

my changes to the .aff files does not seem to help at all. At my sample book i posted above, in the following sentence, the word keinen changes to misspelled, but is correctly spelled:
Da gab’s keinen!

[Doitsu]

@AnselmD: Since Sigil uses the same dictionaries as LibreOffice, I just tested the text from your test epub with curly and straight apostrophes with LibreOffice and in both versions the contracted words were flagged as typos.
I.e., the German dictionary designers most likely didn't define the contraction 's for verbs.
You'll have to add these contractions, which are mostly found in colloquial German, to the default user dictionary.
BTW, I got the same results with Calibre Editor.

Originally Posted by AnselmD

At my sample book i posted above, in the following sentence, the word keinen changes to misspelled, but is correctly spelled:
Da gab’s keinen!

I wasn't able to reproduce this error with the default de_DE dictionary and LibreOffice. Keinen was only flagged with non-German spelling dictionaries.

[KevinH]

With the OLDSPELL it has the wrong WORDCHARS for handling anything related to possesives or contractions:

WORDCHARS ß-.'’

It is hard splitting words after every (')

This makes no sense to me and the modern version of the dictionary uses the following:

WORDCHARS ß-.

I think that should be fixed in the OLDSPELL version but I do not know the language well enough to know. Doitsu, what happens in your testing if that is removed from the WORDCHARS, are correct german contractions handled properly then?

Thanks,

KevinH

[KevinH]

Hi Doitsu,
FWIW, I have grabbed the newest modern de_DE dictionary and converted it to utf-8 with python3 reading it as binary and re-encoding to utf-8 and writing it out as binary with no errors reported. Its aff file does not use either the ICONV or OCONV which would certainly impact if smart single quotes suggestions are generated by the dictionary or only dumb ones. Do you want me to add an iconv and oconv to force suggestions to smart single quotes too?

I will use this version to update the modern German dictionary once you let me know about the preferred iconv oconv settings.

Thanks,

KevinH

[AnselmD]

FYI:
I installed the hunspell command line program for cygwin, in the manual exists the following option:

--check-apostrophe
Check and force Unicode apostrophes (U+2019), if one of the
ASCII or Unicode apostrophes is specified by the spelling dic‐
tionary, as a word character (see WORDCHARS, ICONV and OCONV in
hunspell(5)).

[Doitsu]

Hi KevinH,

Originally Posted by KevinH

With the OLDSPELL it has the wrong WORDCHARS for handling anything related to possesives or contractions:

WORDCHARS ß-.'’

If understand the definition of WORDCHARS correctly, the characters in this list are special characters that otherwise would be considered word delimiters. If that's the case, the German WORDCHARS list, should be identical to the English WORDCHARS list, because the 1996 spelling reform officially sanctioned the use of English style possessive constructions with straight and curly apostrophes in German. (The pre-1996 spelling rules only allowed straight and curly apostrophes in contracted words.)

I.e., WORDCHARS ß-.'’ should be OK for German, because properly spelled German words might contain full stops, hyphens and straight or curly apostrophes. They also may contain the ß character, but only in the middle and at the end of words.
I'm a bit puzzled as to why the dictionary deverlopers specifically added the ß character, but not the other German umlauts (äöü).

However, when I added WORDCHARS ß-.'’ to a utf-8 version of the latest de_De_frami.aff file, it appeared to have no effect, because contracted words such as gab’s were still flagged as typos.

Gab, which is the first part of gab’s (= gab + es), is in the spelling dictionary, which you can test by inserting a space before the apostrophe. I'd have expected the spell checker to ignore this word because gab is in the dictionary.

Since German words may contain curly apostrophes, the German affix file should also contain:

ICONV 1
ICONV ’ '
OCONV 1
OCONV ' ’

However, users, who want to save curly apostrophes in user dictionaries, would have to comment out this section.

D.

[AnselmD]

Originally Posted by Doitsu

@AnselmD:
You'll have to add these contractions, which are mostly found in colloquial German, to the default user dictionary.

Yes, but this does not work. I can add them, but they are still recognized as misspelled.

[AnselmD]

Originally Posted by KevinH

With the OLDSPELL it has the wrong WORDCHARS for handling anything related to possesives or contractions:

WORDCHARS ß-.'’

It is hard splitting words after every (')

This makes no sense to me and the modern version of the dictionary uses the following:

WORDCHARS ß-.

I think that should be fixed in the OLDSPELL version but I do not know the language well enough to know. Doitsu, what happens in your testing if that is removed from the WORDCHARS, are correct german contractions handled properly then?

Thanks,

KevinH

I checked my text with the hunspell command line tool of cygwin. I stored the text as utf8 (using pspad/notepad++). I used the OLDSPELL .dic and .aff files. The words with apostrophe are shown as misspelled, i can add them to the users dictionary and afterwards they are shown as correctly spelled.
This should be the correct behavior.

[AnselmD]

Originally Posted by Doitsu

@AnselmD: You'll have to add these contractions, which are mostly found in colloquial German, to the default user dictionary.
BTW, I got the same results with Calibre Editor.

If i add the words to the user defined dictionary in libre office (using the OLD_SPELL dictionary), they are still misspelled. I will check this with the default German dictionary later. Does anyone manage to add them, so they are not misspelled?