Sigil 1.5.1 too slow in my computer

[KevinH]

I can not reproduce this with Sigil master at all. I will check with stock version 1.5.1. But from just looking at your timings the issue does not appear to be search speed at all and instead seems to be related to the time it takes to open a new chapter and scroll to that position?

Are your timings with Preview open and showing or hidden?

[KevinH]

I tried with Sigil-1.5.1 stock for macOS and I can *not* reproduce this either even with Preview open and HighLight Misspelled words turned on in Sigil Preferences SpellCheck.

Now all of my testing was done after opening the test epub and saying yes to removing the unmanifested file epub/onix.xml and after accepting the to fix the files since they are not well-formed (they are missing doctypes).

Chapter 12 opens almost instantaneously. So does starting with title and nothing else open in tabs and searching for cocaine-bottle.

So whatever is slowing down your machine is very much specific to your platform or system.

So something is making it very slow to open a new file based on your timings.

Since Sigil-1.3.0 and Sigil-1.5.1 share most of the same Qt and Python, that can not be the cause of the problem.

My guess is you have a Windows Anti-virus program that is watching for every new file created in the system temp folder (which is what Sigil does when you open an epub) and scanning it before allowing it to be opened. The Anti-virus program probably accepts Sigil-1.3.0 as okay and exempts it but for some reason has not done the same for Sigil-1.5.1.

Have you tried telling your anti-virus program to exempt Sigil-1.5.1 from its scanning?

[KevinH]

Will people on Windows and Linux please try Selenia's test case and see if you can recreate the timing issue?

I just can not with macOS. It works the same on Sigil-1.3.0, 1.4.3, 1.5.1 and master. It never takes even a second to load and display Chapter 12 of that epub and Searching is just as fast as well.

Please let me know just in case this problem seems to be specific to Windows.

[DNSB]

Originally Posted by KevinH

Will people on Windows and Linux please try Selenia's test case and see if you can recreate the timing issue?

I just can not with macOS. It works the same on Sigil-1.3.0, 1.4.3, 1.5.1 and master. It never takes even a second to load and display Chapter 12 of that epub and Searching is just as fast as well.

Please let me know just in case this problem seems to be specific to Windows.

I tested the file with Sigil 1.5.1 x64, Sigil 1.3.0 x86 and Calibre 5.14 x64 (clean installs of all 3 in a single Windows 10 Pro 20H2 x64 VM running under VMWare Workstation Pro 16.1) I could see no noticeable speed differences between using any of the three programs (tested Sigil with Preview on and off). I also tested with Sigil 1.5.1 and Calibre 5.14 on the real machine and again no noticeable speed differences seen. Just for the heck of it, I also used a Big Sur VM and again no noticeable speed differences seen.

The laptop used has a i9-10885H Processor with vPro™ (2.40 GHz, up to 5.30 GHz with Turbo Boost, 8 Cores, 16 Threads, 16 MB Cache), 64GB RAM and a 2TB NVMe PCIe 3.0 drive. What Lenovo calls a mobile workstation. Antivirus is Windows Defender. The VM had 8 cores, 16GB of RAM and a 256G system drive.

[KevinH]

@DNSB,

Thank you for testing and reporting back.

Wow, that is a monster fast system!

I think I will use your system specs when I plead my case to my wife for a new system! Since I am now retired, my chances are not good


Perhaps someone with more modest hardware (say an i5 and 16gb of ram or less) with Windows could give it a test?

[DNSB]

Originally Posted by KevinH

@DNSB,

Thank you for testing and reporting back.

Wow, that is a monster fast system!

I think I will use your system specs when I plead my case to my wife for a new system! Since I am now retired, my chances are not good

Perhaps someone with more modest hardware (say an i5 and 16gb of ram or less) with Windows could give it a test?

Not something I would have purchased for myself but my employers paid for it. It makes creating and testing VMs before using them on our ESXi hosts much more convenient. The key was asking for an even better system and then backing off to the current system -- $5400 Cdn sounded so much cheaper than $9400 Cdn.

[Turtle91]

Originally Posted by DNSB

...The key was asking for an even better system and then backing off to the current system...

Yuppers! I use that technique on my wife all the time... hit her with the $15,000 dream system, then become "reasonable" and "settle" for the system I was looking at in the first place!


Unfortunately, I think she is catching on to my techniques...

[JSWolf]

Later this year, I'm going to be getting a new computer. I'm thinking of one of those Intel mini-systems.

[DiapDealer]

Using the tests described (and the epub provided) by @selenia on my i3 8Gb laptop, I am able to experience a fairly noticeable difference in speed between 1.4.3 and 1.5.0. The times may not be exactly the same, but everything coincides with their experiments.

It doesn't seem to be dependent upon the tag highlighting or spellcheck settings, turning those off/on seems to make no difference.

What I DO notice right away when opening the file in question (or just about any file in any epub, really) is that starting with Sigil 1.5.1, Preview seems to be slowly "double-pumping" when opening a file: I see the file load, then I see some css apply, then both of those things happen again (when opening any of the epub's files). I see this "double-pump" occasionally on older versions of Sigil as well, but it happens much more quickly, so it's not as pronounced.

The bottom line is that Preview seems to be loading/rendering more slowly in 1.5.1 compared to older versions of Sigil (at least on Windows with hardware that's not supercharged). The good news is that closing Preview makes these search experiments happen pretty much the same speed across all the versions of Sigil tested (1.3.0, 1.4.3, 1.5.1, and a build from github that's only missing the last two commits).

I still experience the same, much slower, double-load of Preview using the master branch of Sigil's github repository (2 commits shy of HEAD), but the new asynchronous loading of Preview means that it no longer affects the search test speeds.

I'm going to try to see if I can grab some video of the double Preview loading going on. But I suck at that sort of thing, so it may take me a bit.

EDIT: Added zip archive of two videos. One for 1.5.1 and one for 1.3.0. MR doesn't allow mp4 file extensions, and I'm not going to take the time to re-encode the videos. Sorry. Just me clicking on the "Find" button with epub in question.

[DiapDealer]

Also note that the process of capturing the video seems to slow things down considerably. So the delays in Preview rendering in my videos are a bit exaggerated, but still representative of the overall difference in load/search times between the versions.

[KevinH]

Hmm ... do you see the double-pump with Sigil 1.4.3 and Preview as well. That is when we introduced the URLSchemeHandler. I wonder if that is the change.

The double-pump is not actually a double-pump but appears as one. Search first opens the target file and then in all versions less than master, it will wait for Preview to load and scroll to its previously stored position, when that completes, CV then moves the cursor to the found text in that file and highlights it. Preview should then happily move to that new location without needing a full reload, Preview should just need to scroll.

But the video clearly shows Preview doing a full reload and then scroll.

Nothing directly changed in any of this code between 1.3.0 to 1.5.1 and this does not happen in macOS (or if it does it happens so fast you can not see it).

If this is not related to the addition of the URLScheme handler after 1.3.0, then we need to look at the signals being emitted by CodeView after its initial load while the scroll to the target text is being done. Somehow that is triggering a complete reload of Preview when only a scroll is done.

Can you see the same thing on Linux?

[BeckyEbook]

Ou, "double-load" or "double-pumping" are good names

I can also see them (also in the latest build).
The fact that this is so can be easily checked in Code Inspector.
Open the "Network" tab and look at the UUID at the end of the file name.
If there is a "double-pump" then it will be seen that the UUID for the same file has changed.

For example, we see the file:

sigil:///T:/TMP/Sigil-aKkZeo/epub/text/chapter-10.xhtml?sigilpreview=7c9ca741-fc4e-406e-ada7-4f815fae0d14

After "double-pump" the address changes and forces the preview window to refresh:

sigil:///T:/TMP/Sigil-aKkZeo/epub/text/chapter-10.xhtml?sigilpreview=27ad347e-09f9-4b0b-ac5d-747e77c4054a

I don't know if this will help diagnose the problem, but "something" forces a refresh and I wouldn't bet it's CSS's fault (although I may be wrong).

---

Regarding the speed tests, and the chapter files in "The Sign of the Four" ebook are too short to spot the speed problem. The differences are noticeable better when we merge all the chapters into one, because then the search for "cocaine-bottle" may take even a few seconds.

I think the reasons are known:
a) underlining misspelled words in Code View now supports two dictionaries (approx. 60% slowdown)
b) new option "Highlight Matching Tags" (approx. 40% slowdown)

When I disable both of the mentioned options 1.5.1 is fast (and the newest build is even faster).
On the other hand – with small files with chapters the slowdown even with the options turned on is slight.

My system: Windows 10 Pro (1909), I5-2500K, 16GB.

PS. Still on the possible slowdown by the antivirus program. This is certainly true, but – importantly – it's not about blocking the application, but rather adding the TEMP folder (where Sigil creates temporary files) to the threat checking exclusion list. I also prefer to note that this is not a safe solution with the TEMP location's default settings.

[DiapDealer]

Originally Posted by KevinH

Hmm ... do you see the double-pump with Sigil 1.4.3 and Preview as well. That is when we introduced the URLSchemeHandler. I wonder if that is the change.

Yes. 1.4.3 shows the double-pump, too. I think it's entirely possible that the URLSchemeHandler might be why there's such a noticeable slowdown in Preview rendering starting with 1.5.1, but the double-pump itself has been around for quite a while.

Forgot to mention that I've excluded Sigil in every way imaginable from Windows Defender on this machine. The process, the program directory, the appdata\local prefs folder, the temp location... the works! But in my testing, I've not really noticed a difference in performance whether Sigil is excluded from Defender's real-time scanning or not.

[KevinH]

Seeing 2 different uuids in Inspector means exactly that that file passed through the URLSchemehandler twice. Which means Preview was told to load twice. But that does not mean the URLSchemehandler is the culprit. Does the problem happen with 1.4.3. If so, it may be that changes in CV or the URLSchemeHandler are the culprit?

[KevinH]

I am going to check the CV code to seewhat underling signals from the TextDocument (held by the HTMLResource in question), triggers a full Preview Update immediately. Just scrolling in CodeView should NOT trigger it.