"document-aware" regex

[mikefulton]

There is a feature that I would love to see added to the regular expressions in Calibre's book editor. It's something I would work on myself if I could find the time, but that doesn't seem to be happening so I thought I'd throw the idea out there to see what people think.

Basically, the idea is to tie the regular expression parsing together with the spelling checker, so that the regex stuff can determine if things are spelled correctly or not.

Here's the use case that prompts this idea. A lot of times when I'm editing an ebook, there are problems regarding hyphenation having been added directly into the text, so you get things like "enter-ing" or "start-ing".

Clearly, with regex, finding a hyphenated word like that isn't a problem. The problem is knowing when it's a good idea to simply remove the hyphen and knowing when the hyphen needs to stay. For example, you wouldn't remove it from "business-like".

So, what if there was a replacement marker that could remove hyphens and then look to see if the resulting replacement was marked as "OK" by the spelling checker?

For example, you'd use "\-1" instead of "\1" (This is just an example and I'll admit I have no idea if it steps on anything in the existing regex implementation.)

If you searched for "\s(.*?-.*?)\s" and it found "enter-ing". If the replacement marker was "\-1" then it would remove the hyphens from the "\1" result to get "entering" and if that passed the spelling check, it would use it as the replacement. If it did not pass the spelling check, then it would use the regular replacement text.

There's undoubtedly a lot of potential for additional functionality once you tap into the spelling checker, but just being able to deal with the hyphens would be a great place to start.

[eschwartz]

Shouldn't this be handled by the spellcheck itself?

In any event, Kovid is working on a macro facility for the editor. That should allow invoking regex and running spellcheck on the results.

[BetterRed]

@mikefulton - have you tried this, select All Words in the Spellchecker and filter with a '-', this was a feature added after the Spellcheckers initial release (filter with '-')

Kovid also added a 'smart' that tests if a hyphen-ated misspelt word can be corrected by removing the hyphen, and if so to bump that word to the top of the correction list. I don't think there's anything you need to do have that.

I found with those two enhancements my 'beef' with hyphens were by and large overcome.

BR

[kovidgoyal]

See my posts in this forum on the function mode for search and replace.

[mikefulton]

Originally Posted by BetterRed

@mikefulton - have you tried this, select All Words in the Spellchecker and filter with a '-', this was a feature added after the Spellcheckers initial release (filter with '-')

Kovid also added a 'smart' that tests if a hyphen-ated misspelt word can be corrected by removing the hyphen, and if so to bump that word to the top of the correction list. I don't think there's anything you need to do have that.

I found with those two enhancements my 'beef' with hyphens were by and large overcome.

BR

If this is something that's been added since 2.0, then no I probably haven't seen it. I wasn't happy with the spelling checker in previous versions and had found other options. I'll have to take another look at it.

[BetterRed]

Originally Posted by mikefulton

If this is something that's been added since 2.0, then no I probably haven't seen it. I wasn't happy with the spelling checker in previous versions and had found other options. I'll have to take another look at it.

@mikefulton - no the features I mentioned predate 2.0 by quite a while. I am not sure if they got a mention in the release notes. But I know they're there, because I was one of those extolling the virtues of having them

BR

[mikefulton]

Originally Posted by BetterRed

@mikefulton - have you tried this, select All Words in the Spellchecker and filter with a '-', this was a feature added after the Spellcheckers initial release (filter with '-')

Kovid also added a 'smart' that tests if a hyphen-ated misspelt word can be corrected by removing the hyphen, and if so to bump that word to the top of the correction list. I don't think there's anything you need to do have that.

I found with those two enhancements my 'beef' with hyphens were by and large overcome.

BR

OK, I've tried what you suggested, and while it works, it's horribly, horribly slow. Hardly any better than simply reading through and editing the text.

The main issue is that you have to manually select the replacement for every unique occurrence.

[BetterRed]

Originally Posted by mikefulton

OK, I've tried what you suggested, and while it works, it's horribly, horribly slow. Hardly any better than simply reading through and editing the text.

The main issue is that you have to manually select the replacement for every unique occurrence.

I suspect my hyphenation problems are not as severe as yours, when Kovid added the 'smart' to offer the de-hyphenated word as the first replacement choice (assuming it is not a misspelling) that fixed a huge number of errors in some 'books' I was working on at the time.

Re the list of hyphenated words, I use it to spot erroneous hyphenations - eg bank-ruptures is not a misspelling but its probably wrong. I paste erroneous hyphenations into my clipboard. Once I've scanned the hyphenated word list I deal with the erroneous sublist I created in my clipboard with regular S&R.

BR