Spellsinger's Blog

[varlog]

This is to document the process of messing up with internals of Sigil - for all ignorants like me who don't know better - to show how it should be done. Or how it should not be done. Whatever. Time will show.

I sort of pledged myself to do something about issue 218 and to entertain/inspire/warn you I will try to describe the process as far as it goes.

First: the development environment. I'm using QtQreator (comes with Qt and has a good online help) in Ubuntu 14.04/64.
So let us start QtCreator.

to be continued...?

[varlog]

I skipped the part about getting sources and dependencies - you can find all necessary information here .

QtQreator is running .
[File]->[Open File or Project] menu shows "Open File" box, let's navigate to source folder (Sigil) and chose the "CMakeList.txt" file there. QtC shows then "CMake Wizard" and wants to know where we shell build Sigil, I just Next'ed its proposal. Now it wants to "Run CMake" and offers us an opportunity to give it some arguments. I use something like this:

-G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX=/path/to/install -DCMAKE_BUILD_TYPE=Debug -DCMAKE_PREFIX_PATH=/path/to/qt/cmake/lib -DPYTHON_LIBRARY=/path/to/python3/library

so in real life it looks like that:

-G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX=/home/varlog -DCMAKE_BUILD_TYPE=Debug -DCMAKE_PREFIX_PATH=/opt/Qt5.5.1/5.5/gcc_64/lib/cmake -DPYTHON_LIBRARY=/usr/lib/python3.4/config-3.4m-x86_64-linux-gnu/libpython3.4m.so

Debug because I expect many many errors . Otherwise it would be Release.
Now QtC runs cmake: no errors so just click on "Finish". QtC parses/indexes.
To check if the project really builds I click on green arrow down left - should build and run Sigil.

it sucks... Embedded Python Error... I had it before but don't remember how I got rid of it (I recreate the process fresh for your sake, my usual environment works as expected).

to be continued...?

[varlog]

I was absent a few days, now I'm back... of course the problem is still there: why my old session works but the new one doesn't? In the same environment?

Have I mentioned before that I have only an hour or so a day to take it on? And a few days in a week (as now!) not even that? So be patient with me... I'm not Kevin .

setext parseropf error -2 traceback Error: traceback report is missing
The program has unexpectedly finished.

???

The only new thing is source fresh from git. There are some others suspects... but the old session is OK...
But the time is up...

to be continued...?

[varlog]

The moment of enlightenment came in the morning, as I rode my bike to catch the 7:26 train. So silly...

My bad habits from old Sigil work flow! This new python Sigil needs to be actually installed for QtC to run it!

So as soon as I could - some 12 hours later - I opened QtC and clicked [Project] item (fifth down on the left - see pictures), there under "Build Steps" [Add Build Step]->[Make] just checked box by "install" item. Just for a good measure "-j 4" by the [Details]-"Additional arguments" for the first build step (make) to use my four cores.

Now click on the green arrow and...
and...
.
.
.

Spoiler:

Sigil runs of course .

Now would be perhaps good time to do some real work?

to be continued...?

[varlog]

A week is gone and all I managed was to compile Sigil - in fully equipped and configured system... doesn't look good, does it? Now perhaps dear Sigil users you start to appreciate having Kevin & Dough on the watch...

Well, my problem is prio 2, just nice-to-have, no deadlines, no stress. Proceeding accordingly.

One thing more has to be done before doing some real work: freezing the original state of the source and starting a new branch. So in QtC we go to [Tools]->[Git]->[Local repository]->[Branches...] and then click on [Add]. In "Add branch" dialog box I name the branch "spellchecker", OK it and the following question "Checkout branch?" answer with yes. QtC shows that is now on the branch [spellchecker].

Now what?

to be continued...?

[varlog]

Bear in mind I know nothing about technology of spellchecking and I havn't used it much in Sigil. And I'm not a linguist. Probably some learning is due.

To find the cure we have to diagnose the patient.

Sigil spellcheck at the moment:
-highlighting of misspelled word can be turned on/off (is checking always on?)
on:
-it seems the spellcheck is triggered by second letter tipped
-it seems one letter words are not checked at all

I think I like the fist one but don't know about the second.
Other spellcheckers I know wait for word break (space, point etc) to trigger spellchecking. Do we need it in Sigil? Is it normal for one letter word not to be checked? Seems so.

The Man Who Started It All defines language like this:

xml:lang="whatever"

applied to appropriate tag - <span> in his case, could be anything I seem to have learned since I've started this. This should be xhtml epub2 syntax.

From what I've learned

lang="whatever" xml:lang="whatever"

seems to be epub3 (x)html5 preferred standard.

I'm thinking about context menu [Set Language] already ...

to be continued... ?

[varlog]

Now Kevin has outlined what's to be done but I have no idea what he's talking about. Well, not quite... but I need specifics.

So into the source.

My way is a little erratic, can't help it. I jump here and there, with every jump getting some knowledge - which hopefully will, in due time, grow into something workable. Let's start with SpellChecker class mentioned by Kevin.

Hey, there you are: src->Misc->(SpellCheck.cpp, SpellCheck.h).

...

Singleton indeed.

NOTE: One could imagine an instance for every language actively used. Just a thought - but I know too little yet to judge the silliness of it. There is this comment there:

//There is a considerable lag involved in loading the Spellcheck dictionaries

Loading new directory by every second word (worst case) versus jumping to (memory consuming) appropriate instance?

For the time being singleton it is.

...

SpellCheck class as comprehended (short version):

Constructor: finds system dictionaries, finds/creates user dictionaries, loads last used dictionary.

NOTE: perhaps the main book language would be the better choice?

Destructor: destructs hunspell and itself.

Functions to remember: all of them of course but setDictionary(QString,bool) is there ready for use.

There is a comment there that clarifies: hyphenation dictionaries are ignored. And then:

// That said, look into getting the raw hyphenation lists that were used
// to generate these hyph_dic files for libhyphen

... but that is not my concern at the moment.

Don't think I've understood it all ...

tbc...?

[KevinH]

Also see Sigil/src/Misc/HTMLSpellCheck.cpp
In particular, the GetMisspelledWords method that walks an html file identifying text and using whitespace and boundary chars to parse it into words to check. This code should be modified to also parse tags and attributes and use that to build a stack that represents current language. The closest code to use as a model is in python called quickparser.py
see Sigil/src/Resource_Files/plugin_launchers/python/quickparser.py.

But any routine would need to written in C or C++ for speed given spellchecking can be done on the fly.

Hope this helps.

[varlog]

Don't be impatient Kevin .

[varlog]

In QtC, when you place the cursor on some function, you can chose from the context menu item [Find Usages] which... finds usages of this function! So, instead of looking up "HTMLSpellCheck.cpp", as suggested, I applied [Find Usages] on class "SpellCheck". I know, I know... but, in my wanderings, I've encountered the "HtmlSpellcheck" class already (I don't write everything here, it's boring enough as it is ): it is a scary one and I'm not up to it yet :|.

So [Find Usages] "Spellcheck":

69 matches found:
SpellCheckWidget.cpp (12)
SpellCheckEditor.cpp (10)
MainWindow.cpp (4)
HtmlSpellCheck.ccp (2) ha!
SpellCheck.ccp (28) I was there!
SpellCheck.h (5) ...and here too!
CodeViewEditor.cpp (8)

Eventually will have to visit all those palaces... but let's look at SpellCheckEditor first...


tbc...?

[varlog]

Had a little more time today...

So src->Dialogs->(SpellcheckEditor.cpp, SpellcheckEditor.h).

This class is responsible for spellchecking cockpit you all know so well (see first picture). The graphical side of it is described in separate file:
src->Form_Files->SpellcheckEditor.ui
in human-readable xml format.

I did a lot of jumping around, met some new classes ("Language" in Misc have some handy functions for one) and old scary acquaintances - but I run away from them again...
It seems to work like this: class MainWindow - which, as you rightly suspect, is responsible for the main Sigil window - initializes, well... dumb class SpellcheckEditor. Dumb because it doesn't know at the moment of creation what book it is supposed to work on. It is important: it means that some useful things to be done have to wait...
MainWindow creates or reads in a new book using the SetNewBook(QSharedPointer<Book>) function, which, among others, tells SpellcheckEditor "Hey, have book". To do this it uses function SpellcheckEditor::SetBook(QSharedPointer<Book>), which is public function of... wait now... SpellcheckEditor! MainWindow, by initiation, first creates Book and than SpellcheckEditor: some time travel here?
BF begins (B is for Brain).

Ok, let it be for now...

I did some things to function SpellcheckEditor::CreateModel(...), wrote some help functions, wrote one fake five-liner (to deliver language of the words), meddled with SpellcheckEditor.ui - QtC has a graphical editor for this, but I prefer to do it manually: this way the diff patch is smaller.

The idea was to create preliminary mock-up of the end product, as a focus for farther thoughts and to get used to Sigil and C++ again. See second picture for results, done on the same sample as the first one.



tbc...?

[varlog]

Eh... it's just an awesome title... and I am considering... but perhaps "The Waste Land" would be more appropriate .

I'm working on SpellcheckEditor cockpit, implementing slowly underlying functions, learning, discovering problems and fighting with them. Did you know that different languages tend to have its own alphabetic order? Despite (almost) the same alphabet?

I certainly won't be implementing that! I mean, if it's not implemented already...?

But surly the different dialects of the same language have to be dealt with.
I decided, for the time being, more or less, to follow this architecture:

MainWindows tells SpellcheckEditor "here, have book!"
SpellcheckEditor asks Book "hey Book, what language you?"
SpellcheckEditor asks Spellcheck "say Spellcheck, what dictionaries you having?"
SpellcheckEditor tells Spellcheck "Spelly, get them dictionarys, ok?"

They are just simple program subroutines, so their conversation is not very sophisticated...

I still don't know how SpellcheckEditor gets his first book. Must be missing something obvious...

See picture for latest cockpit.


tbc...?

[Tex2002ans]

Originally Posted by varlog

Did you know that different languages tend to have its own alphabetic order? Despite (almost) the same alphabet?

More about this topic can be read here:

https://en.wikipedia.org/wiki/Alphab...ic_conventions

I initially ran into this when researching Indexes earlier this year.

Certain languages might treat accented characters the same as unaccented, some might prefer to treat accented characters as completely separate letter (and might mix them between letters or sort them at the very end of the alphabet).

Very interesting stuff when you come from an English-only mindset!

Originally Posted by varlog

I certainly won't be implementing that! I mean, if it's not implemented already...?

I assume Sigil currently just sorts by Unicode position.

In the Wikipedia article linked above, one of the linked articles leads you to the "Unicode Collation Algorithm":

http://www.unicode.org/reports/tr10/

which goes into all the sorting detail you would ever need.

[KevinH]

FYI: Qt has a locale aware comparison function for handling sorting in different locales.
Also Please try to minimize required changes. We don't need the entire spellcheck code rewritten from scratch, we just need to figure out and carry along current language for each word. Thus my earler comment about parsing.

[varlog]

Originally Posted by Tex2002ans

...

In the Wikipedia article linked above, one of the linked articles leads you to the "Unicode Collation Algorithm":

http://www.unicode.org/reports/tr10/

which goes into all the sorting detail you would ever need.

eh... thanks... looks impressive...

Originally Posted by KevinH

...
Also Please try to minimize required changes.
...

I will keep this in mind.