Memory use and sluggish response

[exaltedwombat]

Working on an EPUB, filesize about 1MB. Almost all text. Not unduly long chapters.

Windows Task Manager shows Sigil using about 140MB of memory.

After working on the book for a bit look what memory use has increased to! Over 2GB.

https://www.mobileread.com/forums/at...1&d=1559867211

And Sigil has become sluggish.

It's not just with this one book.

What should I investigate? Thanks.

Sigil 0.9.13
Windows 10
i7 24GB

[KevinH]

Try turning off antivirus and telling Microsoft to whitelist the Sigil folder.

[KevinH]

Also see this post about disabling CFG just for Sigil.
https://www.mobileread.com/forums/sh...&postcount=110

[lumpynose]

It gets sluggish on me as well. When it gets that way double clicking no longer selects words. I haven't paid attention to its memory usage because I have plenty of ram. But when it gets sluggish on me I save and quit and reopen the epub. The epub I've been working on is currently 809 kb; it's a collection of short stories by Max Brand.

[theducks]

For me, it is leaving the Preview pane open (my normal usage) that speeds this slowing down, happening up. I can run longer before it happens, If I don't use the Preview much.
The only cure is ti Exit Sigil.

Windows 'fixed' something :/

[KevinH]

Did you disable Windows CFG for Sigil? Are you running the very very latest version of Windows 10? Did you whitelist/remove Sigil’s program folder from Windows antivirus/defender?

[theducks]

Originally Posted by KevinH

Did you disable Windows CFG for Sigil? Are you running the very very latest version of Windows 10? Did you whitelist/remove Sigil’s program folder from Windows antivirus/defender?

I thought I had disabled CFG for sigil (done now). I really don't like the idea of excluding Program Folders (this is NOT a reflection on the development team). I have no problem on disabling on normal ( but not the download folder) User documents. Once the download has been vetted, I feel real time checking is not needed (I still leave those enabled for system scans).

Call me paranoid , but in almost 40 years of computer use, I have only had 4 infections (And one deployment was triggered by the A/V Check of a download. The A/V software was FIX ED that flaw that day) Another was a 0 day virus

[KevinH]

You may have to change your thinking a bit for Windows 10. If the AI (which is not smart) in Windows 10 defence does not like something a program is doing for any reason it will start to increase scans of generated files, slowing it down and greatly hurt performance. Something as simple as compiling a python py file, or creating a bunch of files in the temp folder can trigger it. So unless you whitelist these specific folders, nothing will help.

In addition other Windows security like CFG has been documented to leak memory and will improperly slow down any program that uses JIT compiled javascript with garbage collection scanning threads that do not start at repeated process memory locations. This is exactly what QtWebKit and QtWebEngine does!

See the thread about what Google engineers found for serious bugs in Windows 10 and earlier, that I posted in the original Release thread.

These are things we can not work around as they are driven by design choices in Qt. Disabling CFG for Sigil and whitelisting it against anti-virus (it is not a web browser) may be the only way around this till Windows wakes up.

If you are truly worried about security, you may want to generate a separate non-admin account to isolate Sigil and your other ebook creation activities from other programs where you can use different security settings more safely.

[KevinH]

BTW, fwiw any system that allows python2 or python3 py files, Perl, or scripting of any sort really, to be run can in no way be secured unless you either trust the authors of the code or eyeball every line of code yourself. This is true of any program but interpreted scripts that are jit compiled can not really be properly code-signed. This includes Sigil’s plugins, Calibre, anything using JavaScript if it can write to the local file system, etc.

This is always a good reason to run anything new or untrusted on a burner non-admin account or in well sandboxed virtual machine first.

Having had a research machine root-kitted over 25 years ago has made me very paranoid!

[lumpynose]

If an epub has javascript in it does sigil run it? I thought I read somewhere that epubs can include javascript. There's also something called ruby but I don't know what it is (and I'm assuming it's not the ruby programming language, e.g., ruby on rails).

I did have myself in a non-admin account with my previous system and it was a pain in the rear having to type the password, even for trivial things in the system control panel. Now with my account being in the admin group I get a pop up window about allowing some admin activity to happen and I only need to click a button. For me clicking a button is less annoying than having to type a password.

I've also now disabled CFG for sigil.

[KevinH]

Currently BookView and Preview will not work without javascript enabled globally in Sigil. When we move to QtWebEngine that drops BookView (see our engine branch on github), we can isolate javascript used to make an application work (ApplicationWorld) from user/epub javascripts (MainWorld) and the provide the user a new General Setting that will allow you to turn off epub javascript inside Sigil.

This is one of the many advantages of us moving away from the older, not fully maintained, QtWebkit to QtWebEngine (ala Google's Chrome) that is properly maintained, with security fixes added in by Qt itself.

Another added advantage is that we can not get QtWebEngine to leak memory even when using embedded fonts, or svg images.

So our upcoming release will be the last to be based on QtWebKit, and the last to support editing in BookView. Soon after our upcoming release we will release the QtWebEngine version of Sigil as a beta along side a beta of a new program called PageEdit that will do much of what BookView did via an Open-With but will run on QtWebEngine.

And that ruby is not related to the ruby scripting language.

[exaltedwombat]

Windows CFG disabled for Sigil. Sigil folder excluded from Windows Security. Reboot

After working an hour on my book memory useage has now topped 3GB. (There's plenty left, but I'm assuming this is connected with the slowing?) Sigil now responding very sluggishly.

Quit Sigil and reloaded book. Memory useage back to 230MB. Response OK.

Arrrrrggggh! I wouldn't mind so much, but when Sigil starts getting slow I find myself dragging blocks of code into the wrong place, applying Clips to the wrong selection etc.

This has only become a problem quite recently. Are we blaming Windows?

[DiapDealer]

Ruby characters

[lumpynose]

Originally Posted by exaltedwombat

Windows CFG disabled for Sigil. Sigil folder excluded from Windows Security. Reboot

After working an hour on my book memory useage has now topped 3GB. (There's plenty left, but I'm assuming this is connected with the slowing?) Sigil now responding very sluggishly.

Quit Sigil and reloaded book. Memory useage back to 230MB. Response OK.

Arrrrrggggh! I wouldn't mind so much, but when Sigil starts getting slow I find myself dragging blocks of code into the wrong place, applying Clips to the wrong selection etc.

This has only become a problem quite recently. Are we blaming Windows?

Try the suggestion from theducks and close the Preview pane.

[KevinH]

I have literally worked editing for hours on many ebooks and have never seen that kind of sluggishness or memory usage and I leave Preview open constantly. But this is on macOS. I have been given one epub to test with that uses many svg image files that does seem to eat memory. But that is a known issue in the latest versions of QtWebkit.

So does your epub have svg image files inside its Images folder?

Does your epub have embedded fonts which seems to cause fontcache leaks on *some* Windows machines?

Are you using a particular feature of Sigil repeatedly that might help us reproduce the issue? Things like Regular Expression Search and Replace, Spellchecking repeatedly, running Reports repeatedly, Using Clips repeatedly, splitting or merging of html files repeatedly, etc? Anything at all that we might try to see if we can reproduce what you are seeing.

Also you might want to ask DiapDealer about trying his AppVeyor automated build of the current master to see if any of the fixes in there actually help (which would explain why we can not recreate the issues).

Please note DiapDealer is not experiencing any of these issues on any of his Windows machines after heavy usage of Sigil. This problem only hits some specific Windows machines and not macOS and not Linux, which is why we think it is a Windows 10 specific issue/bug.

A set of detailed steps to recreate this sluggishness/extreme memory usage with a public test epub (such as our own User's guide) would be really helpful in tracking down if this is a Windows10 issues or something else.