Sigil running multiple instances

[KevinH]

Originally Posted by DNSB

For what it may be worth, I am still seeing the Debug: View Preview Cache Clear failed - deadline expired on the code I downloaded from GitHub this evening.

Wow, that is not good news. The debug log clearly shows that unique names are being created that should prevent any locking issues. It could be some sort of race as DiapDealer said.

We should probably go back to the off-the-record approach to verify it does work for all as creating named storage but deleting it completely upon exit is effectively just doing off-the-record anyway.

I will revert back to a cleaned up off-the-record approach and push it to master. Hopefully that will work for everyone.

[KevinH]

Okay, I reverted to full off-the-record profile just to make sure it really does work for everyone (ie. no deadline expired messages).

I have pushed this to master.

@DNSB and DiapDealer, when you get a free moment would you try launching multiple instances of Sigil (after starting with a fresh debug log file!).

I am hoping this will work for both of you.

Thanks.

[DiapDealer]

The off-the-record approach still works for me without error in Windows 10. No 'deadline expired' messages logged and multiple instances still work. The cache path is empty in the log file, and no cache folders are being created in either the Local or Roaming locations. But I assume that's expected with OffTheRecord cacheing? As I understand it, OffTheRecord is memory based.

I've attached the log file.

RE the named profile strategy:

I found this bug report: https://github.com/martinrotter/rssguard/issues/1496

Which leads to the following Qt bug report: https://bugreports.qt.io/browse/QTBUG-129620

The gist is that the very act of creating a named profile on Windows creates an empty directory in APPDATA/Roaming before the correct data path is even able to be set.

It shouldn't be too hard to clean those paths up, but we need to make sure that there are no problems with running multiple instances on any Windows machines if we ever go back to using that approach.

[KevinH]

Great! Thanks for testing that. Now if DNSB can confirm no expired messages then we are good to go in master.

As soon as we move to Qt 6.9.x then we can revisit things.

[DiapDealer]

I also tested on a physical Windows 11 machine for good measure and got the exact same results.

[KevinH]

Originally Posted by DiapDealer

I also tested on a physical Windows 11 machine for good measure and got the exact same results.

That is good. I have been reading up on how Windows handles disk based web caches. It seems Windows has a special security feature that marks everything in the web cache folder and the folder itself (that can be accessed across multiple threads) as not being executable ever. That makes good sense for a web cache. But a folder needs executable permission to write or change any files in it.

So each instance keeps a list of its own cache folders and gives its webengine process permission to write it it. So clearing a cache folder from one external process that does not have the needed permission will fail. This is something specific to Windows.

I think clearing the cache silently was ignored for any process without permission in older Qt/Sigil but since Qt 6.7 we need to wait for the cleared done signal which revealed the problem.

MacOS shares a single cache across multiple mainwindows all part of the same process, so no problems.

I have no idea how it works with Linux and multiple instances of Sigil. Perhaps all QtWebEngine processes share the same group and can access any cache?

I also have no idea how a new process takes ownership of an existing web cache. The new QWebEngineProfileBuilder forces unique names so how on earth can they take ownership of a web cache that already exists (the whole point of disk caches) even when there is only one instance of the app running.

We ended up wiping all disk caches out to prevent folder proliferation.

I do not think the Qt WebEngine devs have fully thought this through.

[DNSB]

Originally Posted by DiapDealer

Well shoot. That indicates there might be a race condition that one of us is losing/winning.

That's possible. My computer is a couple of years old but still decently fast.

I'm building Kevin's latest update and will report back the status of the issue when it's done.

Finished build and Edmonton/LA overtime around the same time.

Loaded 4 epubs and no errors showing in the debug log. Repeated 4 epubs using Open With plugin in calibre and no errors.

[KevinH]

Wonderful! Thank you! So we have a working fix and it is in master now.

We can revisit this for a future Qt release of Qt with the new QWebEngineProfileBuilder class in 6.9.x with x>1 as Qt 6.9.0 has broken things for many graphics cards.

[DiapDealer]

Originally Posted by DNSB

That's possible. My computer is a couple of years old but still decently fast.

I'm building Kevin's latest update and will report back the status of the issue when it's done.

Finished build and Edmonton/LA overtime around the same time.

Loaded 4 epubs and no errors showing in the debug log. Repeated 4 epubs using Open With plugin in calibre and no errors.

I'm curious about the previous errors you were still getting (when I wasn't) with the disk cache approach. Were you installing Sigil for everyone (with elevated privileges), or just for your user? That might explain why we were seeing different results.

No big deal any more, obviously--and I can test my theory when I get near my development box again--I was just looking for something that might explain it in the meantime.

Our current solution works with either type of install, so it's not really that important.

[DiapDealer]

Originally Posted by KevinH

That is good. I have been reading up on how Windows handles disk based web caches. It seems Windows has a special security feature that marks everything in the web cache folder and the folder itself (that can be accessed across multiple threads) as not being executable ever. That makes good sense for a web cache. But a folder needs executable permission to write or change any files in it.

So each instance keeps a list of its own cache folders and gives its webengine process permission to write it it. So clearing a cache folder from one external process that does not have the needed permission will fail. This is something specific to Windows.

I think clearing the cache silently was ignored for any process without permission in older Qt/Sigil but since Qt 6.7 we need to wait for the cleared done signal which revealed the problem.

MacOS shares a single cache across multiple mainwindows all part of the same process, so no problems.

My Windows user has full permissions to the location where we were writing our cache files in Sigil's prefs directory: C:/Users/<USER>/AppData/Local/sigil-ebook/sigil/

I'll have to fire up an older build and see if that holds true for the cache directories that the qtwebengine process was creating there previously.

Originally Posted by KevinH

I have no idea how it works with Linux and multiple instances of Sigil. Perhaps all QtWebEngine processes share the same group and can access any cache?

I also have no idea how a new process takes ownership of an existing web cache. The new QWebEngineProfileBuilder forces unique names so how on earth can they take ownership of a web cache that already exists (the whole point of disk caches) even when there is only one instance of the app running.

We ended up wiping all disk caches out to prevent folder proliferation.

Not sure. I'd have to be home to check, but maybe suid or sgid or sticky-bit applies to the cache files created by qtwebengineprocess?

[DNSB]

Originally Posted by DiapDealer

I'm curious about the previous errors you were still getting (when I wasn't) with the disk cache approach. Were you installing Sigil for everyone (with elevated privileges), or just for your user? That might explain why we were seeing different results.

The Sigil installer I build pops up the UAC box with a "Do you want to allow this app from an unknown publisher to make changes to your device?" message. Otherwise, no admin rights seem to be involved.

Otherwise, I checked and my user has full rights on the sigil-ebook directories in both Local and Roaming.

[KevinH]

FWIW, I found where in the qtwebengine code all the missing dictionaries warnings are being generated.

Even if you use setSpellCheckEnabled(false) they still search for the damn dictionaries and produce meaningless warnings which they explicitly turn on. This is silly. You obviously do not need the dictionaries if you are *disabling* spell checking.

Unfortunately fixing it requires patching qtwebengine. Even if we do not use it for the upcoming release of Sigil we can add it into our build process for the next Qt release.

[DiapDealer]

I was under the impression that spellchecking was disabled by default unless explicitly set to true in the QWebEngineProfile. So they pop warnings regardless of whether we do nothing, or if we explicitly try to disable it? That's pretty silly.

[DNSB]

I can see some dev going "Just in case someone wants to turn spellchecking on, we want to make sure the dictionaries are available". For now, the directory copied from the Page-Edit install directory to the Sigil install directory keeps me from seeing those error messages.