Sigil-2.7.0 Released

[KevinH]

I am not sure how Kovid's approach works from eyeballing the code.

So I am not sure if Kovid's approach can be used with the >= Qt 6.9.X QtWebEngineProfileBuilder approach we now use. As we could set each cache location for every instance of Sigil's QtWebEngine to use a unique folder name, but a big clean up afterwards would be needed.

Does file contention for cache files only happen in specific folder locations on Windows?

[KevinH]

Originally Posted by DiapDealer

The memory-only caches definitely eliminated the delays for me. I wonder if we could track down which QtWebEngine disk cache is leaking through our memory-cache only strategy for subsequent Sigil instances and patch it on our Windows Qt?

https://github.com/kovidgoyal/bypy/b...qt_base.py#L65

The second and third instance of Sigil should, according to Qt docs return a nullptr when trying to use a named cache folder that is already employed by an earlier version of Sigil.

This must be failing.

The code is here:

    QWebEngineProfileBuilder pb;
    pb.setCachePath(PreviewCachePath);
    pb.setHttpCacheMaximumSize(0); // 0 - means let Qt control it
    pb.setHttpCacheType(QWebEngineProfile::DiskHttpCache);
    pb.setPersistentCookiesPolicy(QWebEngineProfile::NoPersistentCookies);
    pb.setPersistentPermissionsPolicy(QWebEngineProfile::PersistentPermissionsPolicy::StoreOnDisk);
    pb.setPersistentStoragePath(localStorePath);
    m_preview_profile = pb.createProfile("Preview", nullptr);
    // handle possible nullptr return by creating a off the record profile
    if (!m_preview_profile) {
        m_preview_profile = QWebEngineProfileBuilder::createOffTheRecordProfile(nullptr);
    }

And here is what the Builder class docs say ...

QWebEngineProfile *QWebEngineProfileBuilder::createProfile(const QString &storageName, QObject *parent = nullptr) const

Constructs a profile with the storage name storageName and parent parent.

The storage name is used to give each disk-based profile, a separate subdirectory for persistent data and cache. The storage location must be unique during application life time. It is up to the user to prevent the creation of profiles with same storage's location, which can lead to corrupted browser cache.

A disk-based QWebEngineProfile should be destroyed before the application exit, otherwise the cache and persistent data may not be fully flushed to disk.

Note: When creating a disk-based profile, if the data path is already in use by another profile, the function will return a null pointer.

Notice the last line.

[DiapDealer]

I think his patch was probably an attempt to centralize the location for all Webengine cache stuff. Prior to 6.9 and WebEngineProfilebuilder, I think WebEngine sprinkled cache folders all over Windows. My assumption was that WebEngineProfileBuilder might still be leaking something to disk (or looking for something on disk) even when we're trying to make subsequent instances of Sigil use a memory-only cache strategy. Which is causing some file lock issues on Windows. I could be all wet though. Kovid's patch might not help us at all with that, but I thought it might help us track down where any such stray disk cache files might be being written by the later Sigil instances.

[BeckyEbook]

FYI:
Commit e33889e – works OK, the preview window changes almost immediately in the second instance as well.

Commit 317efbb – the preview window opens with a delay in the second instance.

[KevinH]

Originally Posted by BeckyEbook

FYI:
Commit e33889e – works OK, the preview window changes almost immediately in the second instance as well.

That e33889e is using full off-the-record caching. Nothing is ever written to disk or remembered.

Originally Posted by BeckyEbook

Commit 317efbb – the preview window opens with a delay in the second instance.

That 317efbb commit still uses the same folder for persistent storage but all caches are just memory caches.

So it is using the same persistent local storage that is causing the issues even with full memory caches!

Wow!

How can there be a file lock contention issue with a memory based cache? Unless that are memory mapping a file to create the cache.

So there is no easy way to fix this unless we can somehow detect if another Sigil process is already running but that is hard without races happening?

Is there any kind of Windows api to check if another instance of your program is actually running without all the potential for races?

Or we go to a multiple-main Window interface like Word, Excel, and MacOS uses?

How would Windows users react to a mutliple-main window approach? It would only impact people who use multiple instances of Sigil at the same time.

Alternatively on windows we could try adding a uuid to each storage name to make it unique but then we would have to clean up and remove that folder making it effectively off-the-record (ie. no persistent storage).

Jokingly ... we could create a Sigil2 app that only uses OTR all the time, and ask the user nicely to run it when you have already opened a version of Sigil. That would go over well!

I swear Windows goes out of its way to make things hard!

Thoughts?

[BeckyEbook]

The Sigil interface is just perfect for me (well, almost), so I can work with files with a smile on my face. Each new version introduces small but valuable improvements, so it gets a little better and better each time.
I can't even imagine what it would look like with multiple main windows, and whether it would be a step backward in development. But who knows? Maybe I'm being too cautious?

As for the use of an additional UUID, I don't know, we can test it.

[KevinH]

The multiple main window approach will look and act almost exactly like Sigil does now on Windows. But when you open a new file, it does not replace the old main window but instead creates a new main window. If you want to close the old window, you can easily do that at any time. You control it.

The key is that there is only one Sigil program running no matter how many main windows/epubs you have open. Just like Word, Excel, and almost all browsers do now on Windows. Thus only one cache (larger though if multiple main windows open) in use at any time. Avoiding this whole file contention issue.

The problem with the uuid approach is that the created folder must be erased on Sigil exit otherwise your disk would eventually overflow, meaning no persistent storage is possible. That means no Inspector settings being saved or remembered for example. That is not good.

[BeckyEbook]

Actually, this is an option I don't like to use (remember, this is my subjective opinion).
I've always used separate windows, so my Word 2010 and Word 2021 create separate windows.

EDIT: Perhaps I am misunderstanding, but if it were easy to switch between open documents, maybe it wouldn't be so bad.

[KevinH]

They are separate Windows, they just share the same application. You can have them both open and easy cut and paste between them just like you do now.

FWIW, I have found a Qt QSharedMemory approach that will safely determine if another version of your app is running so that the right cache type can be chosen. I will look into that approach first.

[BeckyEbook]

I'm keeping my fingers crossed!
When I see a new commit, I'll test it.

[KevinH]

Okay, I have now just pushed to master an approach that tries to test if a version of Sigil is already running and if so, create an off the record type cache while the first instance gets a normal disk based cache.

It changes:

src/MainUI/MainApplication.h
src/Misc/WebProfileMgr.cpp
src/main.cpp

so you may want to make a clean rebuild of Sigil due to the header file changes.

I have also pushed this commit using deploy to master in case DiapDelaer gets a chance to try it too.

I also have my fingers crossed.


Update: I also added in some debug that shows if that Sigil is the first instance or not using QDebug

Here is what it shows on MacOS (with multiple epubs / MainWindows open):

Debug: WebProfileMgr:: first instance of Sigil: true
Debug: first instance Preview profile: QWebEngineProfile(0x600003c0e420)
Debug: first instance Inspector profile: QWebEngineProfile(0x600003c1e840)

So if you set an environment variable to create a debug log file for Windows, we can see if properly detecting the first and second instances too.

Whoops! forgot to actually push it to master. Now I have

[KevinH]

For those who use multiple instances of Sigil at once on Windows and had issues with Preview slowdowns, please give the following test build a try.

Sigil-Test-AdaptCacheOnFirstInstance-Windows-x64-Setup.exe

It is Sigil 2.7.0 but adapted to test if it is the first instance of Sigil actually running or a later instance. If the first, it nicely uses the normal Disk Caches and will remember things. If it is the second or later, it will use OFF-THE-RECORD caching that should not have problems or interfere with the first instances disk caches - BUT this needs to be tested.

You can download that build from my personal repo which can be found here:

https://github.com/kevinhendricks/Bu...nMac/releases/


If it fixes your issue with multiple instances of Sigil on Windows, feel free to keep using it as it official 2.7.0 with some extras added.

[BetterRed]

Windows 10 (extended license) up to date - no delay in showing Preview in 2nd and third instances of Sigil

BR

[DiapDealer]

It worked in my brief testing on my Windows 10 laptop. I haven't had a chance to check out the code changes yet.

[KevinH]

Great! This change should be the best of both worlds: Sigil's Preview will keep and use local persistent storage and a disk cache for Preview just like normal. Launching the second (or later) instance of Sigil while the first is still running, will use an Off-The-Record cache for its Preview instance to prevent cache file lock contention.

For people who typically use one instance of Sigil at a time there are no real changes.

Thanks for the bug report with instructions to recreate the issue on Windows and all testers.