![]() |
Sigil 1.5.1 too slow in my computer
Hello.
The new version 1.5.1 runs very slow on my computer (Windows 10 Home, Intel i5-3210M, 2.50 GHz, 6 GB RAM). It shows the changes with a lot of delay in the WYSIWYG panel. I am back to 1.4.3, which works fine for me. Please can you tell me if there is something wrong with my computer or it is just too old? Thanks. |
Assuming this is with the exact same epub, nothing much related to speed has really changed between those two versions. And in fact 1.5.1 is, if anything, just a bit faster.
My guess is you have a anti-virus program that is set to scan every file created in tmp and it sees Sigil-1.4.3 as safe and Sigil-1.5.1 as unsafe and is scanning every file is opens or tries to load which hugely slows down Sigil. If there is some way of telling your anti-virus program that Sigil-1.5.1 is safe and should be left alone, you could try that. Better yet, as a test disconnect your computer from the internet briefly and reboot with any added anti-virus disabled, and the do the speed test. You should see that for the exact same epub, Sigil-1.5.1 is slightly faster. |
Perhaps other Windows 10 users can say whether they are seeing or have seen anything similar. That said, no one else has complained about a big slowdown from 1.4.3 to 1.5.1 on Windows.
On macOS, actual large single file opening timings show Sigil-1.5.1 is a bit faster at loading xhtml that really should be split into separate chapter files. |
Thank you for your answers. The antivirus is the default Windows antivirus. I think I have solved the problem by changing the power plan to "Balanced (recomended)". But I still don't understand why this was not needed with Sigil 1.4.3.
|
Quote:
BR |
Forgive me, but what is a "power plan" and how or why is it slowing down one version of Sigil (1.5.1) but not another (1.4.3) when they use the exact same Qt and python libraries.
|
Makes no sense to me. Same Qt, same Python in both versions. Built with the same Visual Studio and the same build environment.
I can't detect any real speed differences between 1.4.3 and 1.5.1 on my i3 2.4Ghz Inspiron with 8Gb of ram. First load of Sigil after a reboot is a bit lengthy for both, but subsequent launches are tolerable. |
Quote:
https://www.tenforums.com/tutorials/...dows-10-a.html These control basics like:
to more advanced things like:
For example, laptops might come by default with "Power Saver" mode to save battery. When plugged in, they'll switch to "Balanced" or "High Performance". Side Note: Can read a bit more technical stuff in the documentation here: https://docs.microsoft.com/en-us/win...wer-management https://docs.microsoft.com/en-us/win...olicy-settings https://docs.microsoft.com/en-us/win...leeping-states Quote:
Quote:
Usually it happens in CPU-intensive things (like games or benchmarks). When the CPU is overheating, the laptop would throttle the CPU. But as you guys have explained, nothing of real difference between those 2 versions... |
Quote:
If the OP's computer is a desktop machine I can't see why a power plan would have the slightest effect on the performance of Sigil. Looking in the advanced power settings there appears to be nothing that would have any impact on performance. As Tex2002ans said it mostly relates to how the computer goes sleep after a set time of inaction, and what happens when you press various buttons. |
Quote:
In general it is a bit slow (it is old), but now it works better and I can use Sigil 1.5.1. Before, the problem occurred only with the WYSIWYG panel open. When I modified a class, this panel would freeze blank for about 5 seconds before reflecting the change. |
No doubt exacerbated by working on epubs that have a single, quite-large, html file which perhaps contains many inline images? That, and a noisy debug build of Sigil (with the debug logfile configured/enabled) are the two most common things that will cause a considerable slowdown in Sigil's Preview load/refresh times.
But the exact same epub should still load/refresh in the same time using Sigil 1.5.1 as it does with Sigil 1.4.3 regardless of any power-management settings. That doesn't mean there's not some specific (and as yet undetermined) set of circumstances that can cause a severe slowdown of Sigil (in any recent Sigil version) on Windows: graphics hardware/drivers, specific versions of Windows 10 and/or Defender definitions, security settings, other programs running, Qt-specific environment variables set at the session-level to support other Qt-based applications, time since last reboot, etc... There's been too many reports of severe slowdowns on Windows 10 (with newer/beefier hardware than the aging laptop I use for testing) reported for me to dismiss them outright. I just know that it's not recent code-changes to Sigil that's the culprit. |
Quote:
BR |
If the slowdown is due to the Preview panel being open, then you are trying to load a much much too large xhtml file.
It is impossible for a browser to render just part of an html file. It literally has to grok the entire file to determine the proper layout and sizing of all of the elements. If you try and keep the entire book as one huge xhtml file, the browser literally has to re-layout the *entire* book for each and every small change. Using Sigil's split-at-marker (after using find and replace to insert them properly) you should split your large xhtml file into separate files, one for each chapter. That is the recommended format for epubs which have to work on large range of devices, browsers, and e-readers. This also speeds up Sigil immensely as having to relayout an entire books's worth of xhtml for every minor change is just plain silly. Whoever recommends people to use one large xhtml file is really off the mark and does not understand anything about how technology or browsers really work. Even Kindle's split the text internally into sections of about 6000 chars or under and tries to show each section independently (as did the old mobi format as well). This was done for speed of loading and compatibility across devices. I have no idea how the newer KFX format handles a large single file but my guess is they are splitting and rendering it in sections or even pages dynamically and not trying to render the entire book at once. |
Sigil preview gets real boggy with large (monolithic -body content file: ~400K>)
IMHO there in NO reason you need to work that way with Sigil. S&R can find & replace over the group of files (or a single) (open) Tabs allow cut and paste moves between sections Split and Join takes care of section boundary issues |
There's been some people mention workflows where they prefer to do certain things with one single file, but their explanations still didn't make much sense to me. I'm not discounting their procedure, I've just never heard anything that would convince me to do it that way. Getting it in bite-sized chunks is step the first whenever I encounter epubs with monolithic html files.
|
For me, I do use one file on occasion. For the most part, when I am editing/cleaning up an Gutenberg sourced epub. Their habit of splitting on file size instead of chapter boundaries makes it easier to merge all the HTML files into a single file and then splitting it on chapter boundaries. Operations on that single file are noticeably slow but that's a temporary condition.
|
Or, maybe the "Samsung optimized mode" tweaks the video driver, IIRC there were/are video driver issues in some versions of Qt.
BR |
Quote:
|
Quote:
Quote:
But the more serious tweak "Power Saver" does is downclocking/throttling more aggressively (especially when on battery). I was doing a few searches to try to figure out what "Samsung Optimized Power" mode does: http://forum.notebookreview.com/thre...lanced.752592/ Quote:
"What causes Windows to freeze for no apparent reason?" NotebookReview Forum: "Anyone here have the NP550P7C 17"?" where users were also having trouble with Samsung's power mode. Now... why it hapens on Sigil 1.5.1 and not 1.4.3... unsure. But I'm still suspecting it has to do with with Samsung's throttling. Question to repilo: Is this happening while on battery? Or is it happening when plugged in too? And did that "Balanced" power mode solve the freezing issue now for Sigil 1.5.1? Does this happen in anything else on your computer? For example, watching a high resolution video on Youtube? Or opening up a very large document in Word/LibreOffice? Quote:
If I remember correctly, if you open super large HTML file:
Note: Although I haven't really messed with large single files in a while... But when I do, I always do my cleanup/splitting in Calibre first. Once it's in bite-sized chunks, then I vastly prefer Sigil. Quote:
For example, merging together a Gutenberg mess or importing a Word "Save As HTML". You initially have your huge file, then do your splitting. But if that initial import is brutal, and Sigil is stuttering/freezing up... it makes it frustrating even if you knew what a subset of the underlying problem was (View > Preview (F10)). * * * Artificial Test Case Take any reasonably large EPUB (>1MB), merge into a single HTML file, then poke away at Code View/Preview. Here's a great example of a 3.9 MB EPUB (~1.4 million words): "The Complete Libertarian Forum" I just tested Sigil 1.5.1 in my VM: 1. Merged all chapters in the EPUB into "Chapter1.html". 2. Clicking View (or anywhere else) instantly froze up Sigil for a while. 3. Once the menu finally opened, even moving the mouse or hovering over menu items caused a huge delay + Sigil (Not Responding). Clicking or doing anything froze it up for another many seconds. Calibre's Editor handles that type of "monolithic mess" at full-speed. * * * Side Note: I also believe this happens when:
Where Calibre copies/pastes in a second (may have to do with syntax highlighting + previewing in a separate thread?)... Sigil completely chugs. I also think that EPUB's a brutal case to test merging lots of files together + updating all the internal links. Here were my rough timings: Calibre: ~2 seconds to merge all files. Instantly able to edit. Sigil: ~60 seconds to merge all files. ~45 more seconds until Code View was even scrollable/clickable. (Preview was closed. Didn't retest with Preview open.) |
I am curious ... is the fast time for calibre to merge, is that done with Preview actively showing the file being merged into, or a different file, or is that with the Preview updating message showing?
A second thread just to load Preview is doable but seems like overkill. But waiting for Preview to load could be handled more asynchronously, as long as it does not interfere with syncing from Preview back to CodeView. I assume calibre is using multiple threads to handle the merging as well. That might make some sense to look into. |
Quote:
I just tested Calibre again, and it looks like it does this: 1. Preview is generated. 2. Keep "Chapter1.html" open as the active tab. 3. Shift-click + highlight all chapter files. Right-Click > "Merge selected text files" Select Chapter1.html bullet. 4. It looks like preview window still keeps the old Chapter1 preview cache (?) displaying there. 5. The instant you click within Code View, the preview window kicks into "Loading preview, please wait..." and takes ~6 seconds to generate a new one. But as I said, the entire time, you're able to edit code, open up menus, etc. etc. all at full-speed. Sigil, at that point, every single thing you click on or do became a slog. Also, as I was typing my post above (~30 mins), I noticed when I clicked back into the merged Sigil window, the entire window re-froze up and went back to "Not Responding" for maybe ~15 seconds. (Preview not open.) Quote:
|
One last question since I do not use calibre ... does clicking in their Preview make CodeView sync to the same place and visa-versa like Sigil does? And if so, what happens if you try to sync from Preview to CodeView when Preview is cached during the merge time window, before the updating message comes up?
|
Also does calibre properly test all files to be merged to check if well-formed *before* starting the merge and giving you a chance to abort before the merge? Does it check for duplicate or missing url fragments in all files and give you a chance to abort before starting the merge?
Right now Sigil is checking for well-formed before, during, and after which seems to be a bit of overkill since testing just before should be enough. That should make for a simple speedup. |
Quote:
A click in Preview jumps you to exact location in Code View. Side Note: Calibre also has 2 helpful buttons below Preview:
(I have whatever the defaults are... both of those buttons are off.) Quote:
When you: 1. Merge 2. Press into Code View. Preview says "Loading preview, please wait..." for a few seconds. Everything works great and is responsive the entire time. When you: 1. Merge 2. Press into Preview. Calibre stuttered a little bit. When I tried to scroll or click around in Preview, it wouldn't work, and neither would Code View (or the entire Calibre window). But within ~6 seconds, it recreated another Preview and everything was back to normal. Quote:
I went into Chapter2.html and busted one of the tables by removing the closing </tr></table>: Spoiler:
and Calibre's merge silently "corrected" + shoved it at the very end of "Chapter2.html"'s code: Spoiler:
so in the final merge, you had: - Chapter1's HTML - Chapter2's HTML --- With busted <table> removed from location, placed at end before Chapter3, and corrected. - Chapter3's HTML Quote:
Like I gave it 3 chapters with the same: Code:
<p class="footnote"><a href="#ft.ast1" id="fn.ast1">[*]</a> [...] </p>Code:
<p class="footnote"><a href="#ft.ast1" id="fn.ast1">[*]</a> [...] </p>And on non-existent/duplicate IDs. No, Calibre doesn't warn about those on merge. It only warns about them when you Tools > Check Book (F7) (similar to an epubcheck) or look in the Reports. Quote:
|
Okay I have Preview being loaded asynchronously in my local tree which helps but the bulk of the time is spent in merge pre-validating xhtml is well formed and that ids are valid.
If we skip those checks (for the time being only) and let gumbo fix things if needed and use multiple threads to update and extract the pieces to be merged, I can cut down that merge time significantly. But that is a bruiser of a test case once merged. It has over 40,000 lines of html code. We synchronously wait for syntax highlighting to be done on purpose to better keep track of actual editing changes vs xhtml syntax colouring changes. Not sure I want to change that given how rare extreme cases like this are and they should be immediately split so should be short lived issues. |
I'm testing a few things that @Tex2002ans described, and I can verify the long periods when Sigil is unresponsive after merging a bunch of files together in the test epub he mentioned above. Many of them I would expect, but the UI interface freezing in general (after the merge is done) seem odd to me (as well as unique unique to Windows, in my experience).
All of my experiments/observations were done with the Preview window closed, with Sigil and it's various files/folders exempt from Windows Antimalware, and with no Power Management restrictions to contend with. The merge of the 60(ish) chapters itself took well over a minute (i3 8Gb RAM). During that time, processor/ram/disk/gpu usage was not particularly high (both cumulatively--for all processes--and for Sigil specifically). One notable thing is that the Windows 10 task manager indicates that power consumption is "very high" during the merge process. The progress indicator for the merge job immediately jumps to 98% and then Sigil becomes unresponsive for over a minute until the merge is complete. After the merge is complete, the UI is unresponsive for a shorter period of time, but lengthy nonetheless. To make matters worse, any Window that covers the Sigil window even briefly (including the task manager which frustrates testing) starts the unresponsiveness of Sigil's UI all over again. From thins point on, I can't really say what might trigger another round of ui unresponsiveness (with the exception of Sigil's window being occluded by another app) when trying to open menus or preferences, but when it happens, the length of time is the same. During the unresponsive periods, Sigil's processor and memory usage spike a bit (but nothing drastic, and not nearly enough to interfere with the running of other applications) and the power consumption indicator immediately spikes to "very high". The gpu % never moves from 0% for Sigil during any of my testing. With Preview closed and no editing being done, I guess I'm not certain why these lengthy periods of UI/menu unresponsiveness are still happening? I get the merge itself being intensive, but once done, why does Sigil not become more responsive when no editing, or rendering is being done? Granted... this is all with the monolithic html tab open that I rarely deal with (by choice) myself, but it would seem that Sigil's Windows performance in general might be improved if we could eliminate some (seemingly) unnecessary reparsing, EDIT: Just to clarify, this is in no way new to Sigil 1.5.1. |
Okay, I have looked into this further. With that testcase merging *all* subsequent chapters into Chapter one.
- a bunch of time verifying all xhtml is wellformed - a bunch if time verifying no ids are undefined ignoring those for now, the rest of the time is split up in 4 pieces - doing the actual merge itself 2 seconds - deleting all of the resources that was merged about 30 seconds - updating all other links in non-merged files and the ncx, about a second The remaining time is spent loading CodeView with the merged file (fast) and then CV running its synchronized syntax highlighting which includes completely spell checking the entire document and then loading Preview (fast since now asynchronous in my local build). So if you use Preferences to turn off spellchecking in CodeView (no highlighting) you see a huge speedup. So could someone turn off Preview, and turn off CV spellchecking in Preferences, and try timing that merge testcase again? How much improvement did you get? |
As for the time to delete the now merged resources, it takes a long time because it requires parsing the opf, updating the manifest, spine, guide, and any nav landmarks (so parsing and recreating the nav each time too), then rebuilding the opf and saving it each time repeated for over 100 resources.
Only the merge case would require deleting so many resources at once, so it was designed for single use and safety not speed. I will look into adding a bulk removal routine to save some time. |
Quote:
Do you have any thoughts as to what is being (re)triggered when clicking on a menu item after everything has completely settled down after the merge and the CV reload? It seems like something is being unnecessarily "redone" at that point. All is quiet and done and then clicking any menu item triggers another lengthy freeze. I can't even get the preferences dialog to fully load after the merge is complete. The widget is blank white, the cursor will spin for a minute or so (titlebar says "not responding") then it will "reset" (for lack of a better word), but soon go back to not responding. Lather,rinse, repeat for about 5 minutes now, and still the Preferences dialog has not rendered anything. EDIT: after about 8 full minutes, the Preferences dialog finally rendered and everything settled down. I'll try to get some tests done with spellcheck disabled. |
Bad news: turns out spellchecking in Code View was already disabled. :(
The same cycle of the Preferences dialog repeatedly doing something for 8 minutes or so before finally rendering happens every time (at least while the new monolithic merged html tab is the one active) after the merge is complete. Sigil debug logging is disabled, by the by. I learned my lesson on that when testing Sigil performance issues on Windows ;) |
1 Attachment(s)
Quote:
I just learned to completely avoid monolithic HTML files + kept Preview closed by default... and now continue to do so out of habit. (Preview + Sigil overall has gotten much faster/better though. :D) Quote:
But that "Very High" makes me think... perhaps the laptop saw that, was in power-saving mode, then decided to aggressively throttle Sigil extra hard to "save battery"? More Testing During/after the merge, Calibre had a few "calibre worker processes" sitting in Task Manager:
Sigil, during these similar steps, used a full core the entire time (~13-15%). When it finally settled down, and became clickable/scrollable... If I left to do something else for a minute, then clicked back into Sigil... it froze and jumped back to a full core for a while. Quote:
https://mises.org/library/man-econom...wer-and-market That was a 1500 page book (each page is marked with an <a id="page_XXX"></a>) with ~6000 links in the Index. (That's the only one I remember off the top of my head where the Index was >300KBs so had to be split. I definitely know I've worked on multiple books with even larger Indexes though, but I don't recall which ones.) Quote:
And Calibre has this nice menu selection where you pick WHICH file you want to merge into: Attachment 186502 |
Quote:
Quote:
Quote:
|
Okay,
I have pushed a bunch of stuff to master for testing to help speedup things: 1. Removed all of the secondary and third tests for well-formed because we check them all in advance now using multiple threads to do the checking. 2. temporarily disabled the id verification step as it really needs to detect if ids are being reused across the the files to be merged (because after merge that could result in duplicate ids). 3. Rewrote the merge code to use multiple threads for speed and to remove redundant checks 4. Added a bulk resource removal capability to FolderKeeper and the OPF to prevent reparsing the OPF a hundred or more times in a row when deleteing them one by one 5. Made loading Preview be asynchronous and no longer wait for loading to complete Now to get the fastest time I do the following: 1) Use Preferences to turn *off* Spellcheck highlighting (red squiggley) in CodeView 2) Use Preferences to turn *off* the newly added highlight tag pair (as that requires reparsing the dom) 3) Preview can now be on or off as it seems to do little to impact speed (interestingly) I added a bunch fo qDebug log messages to show what is going on during the merge so that timings can easily be done when in a debugger The results show the bottle neck is now loading the over 43000 lines of Code into CodeView and then synchronously running the SyntxHighlighter. The syntax highlighter has its own thread but still consumes much time since we wait for completion. The only way to speed up editing any further is to disable syntax highlighting which would need a Preferences setting as well which we do not have (yet). With this in places, what took over 2 minutes on my machine now takes about 15 seconds or so. Any other speedups would come from completely redesigning how we do syntax highlighting which is not something I am prepared to do given the workaround above. |
It seems that syntax highlighting delays things only for initial launch and when CodeView is reloaded.
The real slowdown is actually caused by MainWindow constantly checking if the cursor is in a tag or not so it knows whether to enable or disable some editing icons and menu items. This actually requires CV to search for and identify all tags in the huge document just to know if the cursor is in a tag which might be inside a comment or a cdata or not at all. It needs to do this just to disable icons and things that could happily be ignored after launch if they can do nothing (ie. just to grey things out). I was able to work around the most commonly employed call "IsPositionInTag()" which tries to determine if a cursor is inside a tag or not and properly handle the case of multiline comments and cdata sections. I put a pre filter to at least check near by to look to see if in a tag or not, and if not, just return, if yes, then do the full parse to verify that. I have pushed this to master as well. So all in all, Merge is now a lot faster than it was, Preview can stay open, but disabling SpellChecking in CodeView and Tag Pair highlighting in CodeView in Sigil Preferences will speed things up considerably when working with huge merged monolithic files. |
Quote:
Quote:
(And I always forget... we typically have quite beefy computers. So our "few seconds" may = huge amount of time on an ancient/slow computer. A few years ago, when I was training a few people to use Sigil, I borrowed a person's super slow laptop. Calibre/KindlePreviewer on my computer would convert in seconds, but took MINUTES on the cruddy laptop. I just couldn't believe it... it was like being stuck in the stone ages. :D) |
DiapDealer,
Give current master a try with that testcase but turn off CV spellchecking and open/close tag highlighting first. I am not seeing those long extra pauses anymore after load but I did see them earlier and really have no way to know what is going on in them unless I am running in a debugger and break in and dump all thread backtraces to see where things are. Please let me know if they help. These changes were done quite quickly so expect some breakage ... KevinH Quote:
|
As I mentioned: the stuttering and freezing I've described (as well as the 8 minute wait for the Preferences dialog to render) are all after the massive merge has been completed and Cod View has reloaded.
I expect a massive merge like that to take some time and resources (and it's not something I'd be doing myself, or recommending to anyone). I'm more concerned with Sigil's erratic behavior when the massive html file is open (and Code View is fully loaded). I realize that editing such a file (especially when Preview is open) is always going to be problematic, but I don't understand the freeze-ups that are happening when navigating Sigil menus (especially trying to launch the Preferences dialog) when no edits are being made and Preview is closed. I haven't pulled the changes and retested yet, so I'll do that now. Hopefully they will have an effect on the slowdowns and freezing/stuttering I'm referring to. EDIT: Whoops! Looks like we cross-posted, there. I'll definitely pull the changes and disable the tag highlighting. Spellchecking has been disabled all along. |
With just the Book Browser and Code View widgets open (and with spell-checking and tag highlighting disabled) the same massive merge takes 1.5 minutes in 1.5.1 and just a hair less than 20 seconds after your changes. That's great!
Unfortunately, the stuttering and freezing continues (as well as the delay when trying to launch/render the preferences dialog afterward with little change that I can detect). But I'm beginning to detect a pattern that can help eliminate the problem (temporarily at least). The issue seems to be that the Code View tab with the new giant html file (from the merge) has the focus. If I can get the focus on the Book Browser widget (by single-clicking one of the files in the tree and waiting for the inevitable busy cursor to stop spinning) then the various menus (including launching the Preferences dialog) are very responsive and don't suffer from the "freezups"... even though the giant file is still the active tab showing in the central widget. I know there's no way to keep monolithic html files from slowing down Sigil in general when editing, I'm just wishing there might be a way to mitigate the effect if the monolithic file's tab is open (and active) but is not being actively edited, scrolled, clicked on, or rendered (Preview closed). P.S. I also think the fact that the drop down menus (as well as the Preferences widget) can partially oclude the Code View tab of the giant html file might be causing signals to fire and reparsing to happen. Trying to launch Preferences on my laptop with the monolithic file open actually seems to cause a loop of some kind that prevents the widget from ever fully rendering. If I shrink the Code View tab so that none of the dropdowns touch Code View (and make sure the Preferences dialog is positioned to not open over top of the Code View Tab), the freezing menus and non-rendering Preferences don't seem to present nearly as much of an issue. |
I may have been lucky enough to just have focus someplace else. Opening Preview should not change your timings by much.
If you can recreate this slowdown in launching preferences on Linux, try running gdb and when the beach ball or hang starts, hit crtl-c to interrupt gdb and get a copy of all backtraces of all threads. Then use gdb to hit C for continue, and wait a second or two and interrupt it again. Then again print out the backtraces of all threads. Find the thread that has not progressed much across the two sets of backtraces (that is actually doing something or trying to). That will help show us what is happening at this time. If you get a chance, zip up the two outputs and e-mail me. I will try the same thing under macOS if I can get the big delay after loading to happen. Once we see where things are stuck, we will have a better idea of how to fix it. |
@DiapDealer
Your idea about focus was right. It seems if CV loses the focus because any other dialog opens over it when you try to close the dialog, it causes focus to be regained which will completely rehighlight the entire document which causes a big delay. I would think that there is no need to rehighlight (synchronously) the entire document when focus is lost then gained. Widget backingstore should take care of that without having to redo the complete highlighting. So I have just pushed yet another change to master to disable this hopefully unneeded re-syntaxhighlighting with every change in focus in CV. With this change, I can open and close dialogs that partially cover CV with no more spinning ball. This is even with Preview open, but spellchecking and open/close tag pair highlighting disabled. Please give it a try. I have my fingers crossed. KevinH |
| All times are GMT -4. The time now is 10:57 PM. |
Powered by: vBulletin
Copyright ©2000 - 3.8.5, Jelsoft Enterprises Ltd.
MobileRead.com is a privately owned, operated and funded community.