Epub Accessibility - Changes for Sigil?

[KevinH]

Yes it is a limit in the pcre2 library. We can up it for Mac and Windows since we have our own pcre2 library in our 3rdparty but we can not force its use on Linux.

For the gory details, see my pm'd response.

We will up this limit for the next release. I just need to remember to push that change tomorrow.

[philja]

Originally Posted by KevinH

Yes it is a limit in the pcre2 library. We can up it for Mac and Windows since we have our own pcre2 library in our 3rdparty but we can not force its use on Linux.

For the gory details, see my pm'd response.

We will up this limit for the next release. I just need to remember to push that change tomorrow.

Thanks for tracking that down, Kevin. Of course, I hit the problem in linux, but it seems rather rare.

[DiapDealer]

Originally Posted by KevinH

Yes it is a limit in the pcre2 library. We can up it for Mac and Windows since we have our own pcre2 library in our 3rdparty but we can not force its use on Linux.

The AppImage uses our bundled and statically linked pcre2; so we'll be alright there. Distro builds will probably still have the limitation though. But then they always have, right?

[KevinH]

Yes, but how many people generate a regex search expression longer than 65535 utf-16 chars? But if you use regex to find and match to exactly an entire big block of text like is being done here it could happen.

[KevinH]

For the record in case anyone else runs into this problem.

Here is the limit in pcre2 that was exceeded and needs to be updated:

The entire selected text - which in my case with your test file was 37051 utf-16 chars long is used to create a regex search expression that is fully escaped and had a length of 48588 utf-16 chars.

Unfortunately that search regex is then "compiled" by pcre2 and then used in the search.

The limit on the size of the compiled search regex is set in pcre2 source in its config file here:

/* The value of LINK_SIZE determines the number of bytes used to store links
   as offsets within the compiled regex. The default is 2, which allows for
   compiled patterns up to 65535 code units long. This covers the vast
   majority of cases. However, PCRE2 can also be compiled to use 3 or 4 bytes
   instead. This allows for longer patterns in extreme cases. */
#ifndef LINK_SIZE
#define LINK_SIZE 2
#endif

And that 48588 char long escaped selected text search expression when compiled exceeded that limit.

If I change this value to say 4, then recompile both pcre2 that is in Sigil's 3rd party and Sigil itself pasting a block of text of that size now succeeds.

This limit is set in our own 3rdparty/cmake/pcre2.cmake file so we can up it easily, but it is a build time change not a run time change as far as I can tell. The default value of 2 is set in the PCRE2 config.h.generic that is used by the buildtime "configure".

I have no idea if Qt's internal pcre2 has increased its limit, or what linux distributions do?

[KevinH]

Update I looked in qtbase dev source and found this:

qtbase/src/3rdparty/pcre2/src/config.h

And it sets LINK_SIZE to 2, so Qt's regular expressions have the same limitation.

And according to the Arch linux package build they just use pcre2's configure with no changes so it uses the default value as well, so it will have that same limitation. So my guess is most if not all other Linux distributions do the same.

[KevinH]

We could always redesign all Sigil's clips to not use regex to do a Find and Replace when text is selected. That could be something maybe for a future release.

[philja]

If you up the limit in your 3rd party pcre2, that will fix the problem for most people. I certainly always use DiapDealers AppImage in linux to keep more up to date than the distro version.

And the use case is, and will probably continue to be, rather rare for most users. I only came across it because I was using the relatively recent Insert > Aria Clip functionality to place a <section> with related types and roles around the whole of a chapter.

And that was only with one unusually long chapter. Which I edited manually when the Aria Clip insert failed.

All in all, not a major catastrophe needing an urgent fix.

[DiapDealer]

@KevinH: will the update to our 3rdParty pcre2 suffice, or should we consider patching Qt's pcre2 as well? We're patching Qtbase on just about everything already, so it wouldn't be a huge deal. But if you you don't think it's necessary, I'm fine.
Actually... I take that back! Patching Qtbase would be a huge deal-breaking proposition for the Linux AppImage build process. Not sure WHAT I was thinking.

[KevinH]

No worries, the use of find and replace is only used for clip pasting and it always uses our internal pcre2 not Qt regular expressions. We do use Qt regular expressions in other places but not in clips. The only way to get this issue to happen is to use a clip with a \1 in it and have selected a *huge* block of text which becomes a huge regex search expression because of the way replacement was always handled (ie. not our code, code we inherited).

So there is no need to rebuild qtbase on Linux.