About saved searches and regex

[Carpatos]

I'm a Sigil user, and I'm testing new Calibre editor in order to evaluate it for using as replacement for Sigil.

About saved searches, I found a bug at the interface. In this window (saved searches), if you have a long saved search, with a very long regex in "Find" field, when you select it at the saved searches list, the main window resizes automatically and becomes too width. If you try to resize the window again, it sometimes disappears, and you must close and reopen the edit program to make it visible again.

And other problem is regex doesn't support PCRE expressions, like Sigil does. PCRE are much powerfull than python's "re" regexps. And there is some features in PCRE with not equivalent in "re". Is there any possibility to improve regexp, maybe with some library (like regex or python-pcre) to get PCRE support in the searches?

Thanks for you program, and your support.

[kovidgoyal]

The only useful feature that exists in Sigil that does not exist in the editor's regex engine is the \U operator, and the editor will be getting a function mode for search and replace that will be far more powerful and general than that.

And note that the editor does not use python's re module. It uses https://pypi.python.org/pypi/regex

[DiapDealer]

Originally Posted by kovidgoyal

The only useful feature that exists in Sigil that does not exist in the editor's regex engine is the \U operator, and the editor will be getting a function mode for search and replace that will be far more powerful and general than that.

I've certainly not found the regex module calibre's editor uses to be lacking any real functionality (with the exception of the \U \L \u \l \E replacement operators you mentioned) compared to Sigil's PCRE engine. I do still miss the \K operator at times, but the addition of variable-length lookbehinds makes it possible to replicate \K's versatility anyway.

[kovidgoyal]

Incidentally, do the case change operators in Sigil's regex engine do an ascii swap [a-z] <-> [A-Z] or do they support full unicode case folding?

[DiapDealer]

They seem to at least do some basic unicode case folding. Don't know if it's full support, though.

In a search for the character 'á', all of the below captures it:

(á) 
(\x{00E1})
(\p{Ll})

and \u\1 changes it to 'Á'

But I've no idea if it's limited to a certain subset of unicode (or if it handles all the ways á can be represented).

[kovidgoyal]

That's better than I expected I always assumed that ICU in Sigil was present only for WebKit, but perhaps the regex engine also uses it.

[Papirus]

I'm an other user of Sigil trying to move to Calibre editor.
I'm testing the regex module because I have more than 300 regex (some of them quite complex) for correcting spanish texts.

As it's refered in the initial post, there is a problem, when length of the search chain is long, with saved searches and warning window sizes.

As I said, I have a lot of of regular expressions so in my opinion it would be interesting the possibility of grouping and nesting them in the saved searches window in order to keep them organized.
Another thing I miss is that when I use a saved search expression will automatically appear in the main window find/search area. Sometimes is necessary to make some changes in one of them and now the only way is opening the saved search window, editing the expressión, copying and pasting and then modify it.
In this find/search area, would be possible to add a "count all" button?

Related with the regex engine, I've realized of two main differences: \K (but I can circumvent it using variable-length lookbehinds) and the conditional structure (?(condition)Then|Else); this one is an important limitation compared with PCRE.

Properties \p are well supported, but \p{Lu} (uppercase letter) and \p{Ll} (lowercase letter) only works correctly if "case sensitive" option is checked (I don't know if this is the expected behaviour). I've tried (?f) with no success.

As is mentioned in thread \U \L \E don't work, but they are Sigil commands no PCRE. Any way, it would be very interesting a similar option in Calibre editor because it's a very frecuent mistake in texts: lower case after a dot, and now replacement is not possible.

In another context, sometimes scanned text includes & shy; (soft hyphen), this is a hidden hyphen that you can't see (at least in Sigil) and the only way to remove it is regex searching \xAD. Here the problem is not with the regex engine but with file preview panel where it appears as a dot. A similar behaviour it's with ​& #8203; (Zero-width space), that is also represented as a dot and it's another hidden character that is used in very very very long words in order to break the paragraph avoiding text exceed the screen boundary in readers. Here \x{200B} regex is not allowed.

Thank you very much for your program and support.

[DiapDealer]

Originally Posted by Papirus

Related with the regex engine, I've realized of two main differences: \K (but I can circumvent it using variable-length lookbehinds) and the conditional structure (?(condition)Then|Else); this one is an important limitation compared with PCRE.

I don't use really use if|then|else regex conditionals myself, but the regex module calibre's editor uses certainly supports them. Probably just a matter of getting the syntax right. For example:

(a)?b(?(1)c|d)

Matches both "bd" and "abc"

Properties \p are well supported, but \p{Lu} (uppercase letter) and \p{Ll} (lowercase letter) only works correctly if "case sensitive" option is checked (I don't know if this is the expected behaviour). I've tried (?f) with no success.

This sort of hung me up for a bit too, but when you think about it ... searching specifically for lower- or upper-case letters is sort of the very definition of "case sensitivity," is it not?: hence the reason for the box needing to be checked. If you need case insensitivity, uncheck the box and ... use \p{L} in your expression instead.

In another context, sometimes scanned text includes & shy; (soft hyphen), this is a hidden hyphen that you can't see (at least in Sigil) and the only way to remove it is regex searching \xAD. Here the problem is not with the regex engine but with file preview panel where it appears as a dot. A similar behaviour it's with ​& #8203; (Zero-width space), that is also represented as a dot and it's another hidden character that is used in very very very long words in order to break the paragraph avoiding text exceed the screen boundary in readers. Here \x{200B} regex is not allowed.

The syntax is different for matching specific unicode codepoints. instead of \x{FFFF} just use \uFFFF. So looking for your & shy character becomes \u00AD and the search for the zero-width space becomes \u200B. PCRE was really the odd man out with the \x{FFFF} sequence.

[arspr]

Originally Posted by Papirus

I'm testing the regex module because I have more than 300 regex (some of them quite complex) for correcting spanish texts.

If you feel like it, I would be really, really glad if you shared the most useful ones.

There's a young (and still short) thread about useful S&R settings in this very same subforum: https://www.mobileread.com/forums/sho...d.php?t=237181

[Papirus]

Originally Posted by DiapDealer

I don't use really use if|then|else regex conditionals myself, but the regex module calibre's editor uses certainly supports them. Probably just a matter of getting the syntax right. For example:

(a)?b(?(1)c|d)

Matches both "bd" and "abc"

This is not the situation.

Imagine a text and that you want to look for roman numbers in order to small caps them.

([ivxlcdm]+) could be a possibility with <small>\1</small> as replacement.

The above expression will return roman numbers as well as different words that are formed with these characters.

I.e. clic, CD, ill, id, lid, livid, isolated letters, etc.

Probably in spanish there are some more frequent: Thousand (mil) and mainly My (mi)

So if I use the above expresion of course I will get romans numbers but hundreds coincidences of non roman numbers too. The only way I know to bypass this is through conditional structure (?(condition)then|else). It would be more or less (in fact is necessary to refine the expression):

(?i)(?(?=[clv]?i?d|[cm]m|[dm]i|clic|lcd|(m|ci)?v?il|[mdcl]|ill|livid) |(?<=\PL)([ivxlcdm]+)(?=\PL))

When the condition is satisfied (common words that are not roman numbers) the yes-pattern is used (we look for nothing: the white space in this example after de condition parenthesis). Otherwise we will look for romans.

[Papirus]

Originally Posted by arspr

If you feel like it, I would be really, really glad if you shared the most useful ones.

There's a young (and still short) thread about useful S&R settings in this very same subforum: https://www.mobileread.com/forums/sho...d.php?t=237181

They are written in PCRE, so they should be adapted to python.

[kovidgoyal]

@Papirus: Thankfully, calibre's editor is written in python so you dont have to create that unreadable gunk to do something like change case for roman numerals. Using the upcoming function mode, all you need is

Find: (?i)\b([ivclm]+)\b
Replace:

def replace(match, context):
    word = match.group()
    if word.lower() not in {set of common words}:
         word = word.upper()
    return word

[DiapDealer]

Originally Posted by Papirus

This is not the situation.

Imagine a text and that you want to look for roman numbers in order to small caps them.

([ivxlcdm]+) could be a possibility with <small>\1</small> as replacement.

The above expression will return roman numbers as well as different words that are formed with these characters.

I.e. clic, CD, ill, id, lid, livid, isolated letters, etc.

Probably in spanish there are some more frequent: Thousand (mil) and mainly My (mi)

So if I use the above expresion of course I will get romans numbers but hundreds coincidences of non roman numbers too. The only way I know to bypass this is through conditional structure (?(condition)then|else). It would be more or less (in fact is necessary to refine the expression):

(?i)(?(?=[clv]?i?d|[cm]m|[dm]i|clic|lcd|(m|ci)?v?il|[mdcl]|ill|livid) |(?<=\PL)([ivxlcdm]+)(?=\PL))

When the condition is satisfied (common words that are not roman numbers) the yes-pattern is used (we look for nothing: the white space in this example after de condition parenthesis). Otherwise we will look for romans.

Sorry. Perhaps I wasn't clear. I wasn't trying to give you a conditional expression that would do exactly what you want. I only intended to let you know that the if|then|else conditional construct is definitely supported by the editor's regex engine. It's going to be up to you to adjust your existing conditional expressions to work with the new engine.

I think the disconnect here is terminology. The calibre editor's regex module supports if|then|else conditionals: that part is not up for debate. I think where you may be running into problems is that it doesn't support conditionals using lookarounds. So instead of a conditional like:

(?(?=regex)then|else)

You might need to employ two opposite lookarounds:

(?=regex)then|(?!regex)else

to achieve the same end.

So in essence, your:

(?i)(?(?=[clv]?i?d|[cm]m|[dm]i|clic|lcd|(m|ci)?v?il|[mdcl]|ill|livid) |(?<=\PL)([ivxlcdm]+)(?=\PL))

becomes:

(?i)(?=[clv]?i?d|[cm]m|[dm]i|clic|lcd|(m|ci)?v?il|[mdcl]|ill|livid) |(?![clv]?i?d|[cm]m|[dm]i|clic|lcd|(m|ci)?v?il|[mdcl]|ill|livid)(?<=\PL)([ivxlcdm]+)(?=\PL)

Ugly ... but doable.

[DiapDealer]

Originally Posted by kovidgoyal

@Papirus: Thankfully, calibre's editor is written in python so you dont have to create that unreadable gunk to do something like change case for roman numerals. Using the upcoming function mode, all you need is

Find: (?i)\b([ivclm]+)\b
Replace:

def replace(match, context):
    word = match.group()
    if word.lower() not in {set of common words}:
         word = word.upper()
    return word

The upcoming function mode sounds like it's going to be terribly useful.

[kovidgoyal]

In fact, just for giggles, since the function mode is actually arbitrarily powerful, you could do this:

from calibre.gui2.tweak_book import dictionaries

def replace(match, context):
    word = match.group()
    if not dictionaries.recognized(word):
         word = word.upper()
    return word

Which even gets rid of the need to define your own list of common words, it will use the dictionary for whatever language is specified in the books opf file (provided of course you have installed such a dictionary).