Indefinite length lookbehind

[ElMiko]

I'm trying to find instances of the following string

said. “

(where the quotation mark is a curly opening quotation mark)

I want to exclude matches where the string is preceded by a word, preceded by a closing curly quotation mark. e.g.

” Jack said. “

My solution for this would be a regex search like:

(?<!”\s\w+?\s)said\. “

However, as soon as I add the quatifier "+" I get the error message for invalid Regex.

I was under the impression that 2.4.2's regex natively allows for indefinite length lookbehinds. What am I doing wrong? Is there some a different syntax that needs to be used for indefinite length lookbehinds?

[Turtle91]

Try giving it a batch of characters to choose from enclosed with square brackets?

(?<!”[\s\w\.]+)said\. “

Although I have a feeling that there is a much simpler way to accomplish what you are trying to do instead of this very narrow regex. What is your end goal??

Edit:
You might even try tokenizing the \. in the negative look behind pattern to catch any punctuation \p{P} instead of just periods.

(?<!”[\s\w\p{P}]+)said\. “

Please note: I’m replying on my phone after spending a very sleepless night with a grandson who is sprouting some molars … please test it and forgive any errors…

[ElMiko]

@Turtle91 - I never cried as a child, and I started reciting sonnets in the natal ward.

Unfortunately, this syntax doesn't work either. As with my attempt, the element that breaks it is the quantifier "+"—basically, the bit of the search that is supposed to be making it indefinite in length!

The problem I'm trying to solve is that the OCR misread many commas as periods, resulting in text like:

He turned as Charles said. “Howdy!"

However, if I just search for "said\. “", I'll get false positives such as

“Let's go,”  Charles said. “I think I'm done here.”

Hence the structure of my search and the negative lookbehind.

[ElMiko]

Hmmm, I found this...

But with all respect to the author, I can't make heads or tails of the explanation... much less how to apply it to anything other than matching the letter "X"...

Just as importantly, I can't even get it to match the letter "X" in any given Sigil file...

[KevinH]

See the pcre2 maintain had to say when he implemented this in 2023 here:

https://github.com/PCRE2Project/pcre2/issues/269

It seems the PCRE2 approach requires a backwards max range and not a +

[ElMiko]

Originally Posted by KevinH

See the pcre2 maintain had to say when he implemented this in 2023 here:

https://github.com/PCRE2Project/pcre2/issues/269

It seems the PCRE2 approach requires a backwards max range and not a +

Afraid this doesn't seem to be working either. I tried:

(?<!”\s\w{1,10}\s)said\. “

and still get the "Regex Valid?" error message.

[KevinH]

What does the exact error message say? Mouse over the find field or valid regex symbol?
Does it show the exact error message? Try a character range not a word range. Did that change the error?

[KevinH]

I checked the pcre2 source for changes and saw this:

Version 10.43 16-February-2024
------------------------------

There are quite a lot of changes in this release (see ChangeLog and Git log for
a list). Those that are not bugfixes or code tidies
...

* Added support for limited-length variable-length lookbehind assertions, with
a default maximum length of 255 characters (same as Perl) but with a function
to adjust the limit.

And the version of PCRE2 in Sigil 2.4.2 is version 10.44 so support for limited length lookbehind assertions should be in Sigil 2.4.2.

Are you using an assertion properly? A more specific error message might help if you can get one.

[Doitsu]

@ElMiko A long time ago I created a throw-away Sigil regex tester validation plugin that should theoretically work for your regex.
After the installation you'll find the plugin under Plugins > Validation > RegexTester. (You'll need to select the "regex" engine.)

In my test case:

<p>Lorem “ipsum dolor” Jack said. “</p>
<p>Lorem ipsum dolor said. “</p>
<p>Dolor amet said. “</p>

it only selected the second and third paragraphs when I searched for:

(?<!”\s\w+\s)said\. “

For more information on the alternative regex Python module see the official documentation.

[ElMiko]

Originally Posted by KevinH

What does the exact error message say? Mouse over the find field or valid regex symbol?
Does it show the exact error message? Try a character range not a word range. Did that change the error?

Please see the attached screengrab

I've tried several permutations of the regex:

\w
\S
\u
\l
\D
[a-z]
.

All show the same error, and return no matches.

@Doitsu - Yeah, I don't know what's going on.

[theducks]

\S is not \s
\S is not a space char

[ElMiko]

I know that, theducks. That's why I used it. Non-space, followed by min/max range, followed by space. But even if it were a mistake, the point is that the STRUCTURE is being interpreted as invalid.

But also, even if it had been a mistake it wouldn't explain why the other variants aren't working either.

[KevinH]

It is our long holiday weekend in Canada, but I finally got some time to test things in Sigil on my only laptop up here at my cottage. It is a pre-release version of the forthcoming Sigil v2.50.

I decided to test the example cited by one of the issues posted at PCRE2 in that link I posted earlier.

In my xhtml file I have:

<p> 0xxxy </p>

And as my find I have:

(?<=0x{1,6})y

And I get no error and it works to find the y in the text.

Would you please try this test with your Sigil 2.4.2 and let me know if you get the same thing? Perhaps there was a bug in PCRE2 10.44 that got fixed in PCRE2 10.45 which is in the upcoming release of Sigil.

I will try Doitsu's test next.

[KevinH]

Okay I tested the following in Sigil 2.50 (pre-release):

The xhtml file was:

  <p>Lorem “ipsum dolor” Jack said. “</p>

  <p>Lorem ipsum dolor said. “</p>

  <p>Dolor amet said. “</p>

  <p> 0xxxy </p>

And the search string I had to modify to make it a finite quantity:

(?<!”\s\w{1,6}\s)said\. “

And when I run it the find only goes to the second and third lines.

So as far as I can tell with these examples, all is working.

But again this version of Sigil has a newer version of PCRE2 (10.45) than the version that came in Sigil 2.4.2 (10.44), so since you are seeing something different I would guess that there was a PCRE2 bug in 10.44 that got fixed.

If it is any help, we are hoping to do final updates of the translations this week and will try to make a full release by next weekend if both of us can work it into our schedules.

If you desperately need something immediately, I can generate a CI build of current Sigil master (it will be missing translations in most languages) and make a link available to you.

But please test your Sigil 2.4.2 build and let us know if it fails these very specific tests (ie. if there was a PCRE2 bug).

[ElMiko]

No rush on my end; I can put a pin in this one until the next release. Thanks, as always, KevinH!