Sigil 1.5.1 too slow in my computer

[DNSB]

For me, I do use one file on occasion. For the most part, when I am editing/cleaning up an Gutenberg sourced epub. Their habit of splitting on file size instead of chapter boundaries makes it easier to merge all the HTML files into a single file and then splitting it on chapter boundaries. Operations on that single file are noticeably slow but that's a temporary condition.

[BetterRed]

Or, maybe the "Samsung optimized mode" tweaks the video driver, IIRC there were/are video driver issues in some versions of Qt.

BR

[DiapDealer]

Originally Posted by DNSB

Operations on that single file are noticeably slow but that's a temporary condition.

Closing the Preview window under those conditions also eliminates the constant refreshing/reloading going on with every individual Code View edit.

[Tex2002ans]

Originally Posted by BetterRed

Or, maybe the "Samsung optimized mode" tweaks the video driver, IIRC there were/are video driver issues in some versions of Qt.

Hmmmm...

Originally Posted by Ashjuk

As Tex2002ans said it mostly relates to how the computer goes sleep after a set time of inaction, and what happens when you press various buttons.

Well, the default Windows 10 power settings pretty much do that.

But the more serious tweak "Power Saver" does is downclocking/throttling more aggressively (especially when on battery).

I was doing a few searches to try to figure out what "Samsung Optimized Power" mode does:

http://forum.notebookreview.com/thre...lanced.752592/

The 'Samsung Optimised' power plan seems to cap the CPU cores to 50% of their regular full clock speed on my NP700Z7C. [...]

There's also these:

"What causes Windows to freeze for no apparent reason?"

NotebookReview Forum: "Anyone here have the NP550P7C 17"?"

where users were also having trouble with Samsung's power mode.

Now... why it hapens on Sigil 1.5.1 and not 1.4.3... unsure. But I'm still suspecting it has to do with with Samsung's throttling.

Question to repilo: Is this happening while on battery? Or is it happening when plugged in too?

And did that "Balanced" power mode solve the freezing issue now for Sigil 1.5.1?

Does this happen in anything else on your computer? For example, watching a high resolution video on Youtube? Or opening up a very large document in Word/LibreOffice?

Originally Posted by DiapDealer

Closing the Preview window under those conditions also eliminates the constant refreshing/reloading going on with every individual Code View edit.

Calibre's Preview doesn't choke as much as Sigil's.

If I remember correctly, if you open super large HTML file:

• Calibre lets you use Code View at full-speed while Preview is chugging away—displaying a temporary "Loading preview, please wait...".
• Sigil's freezes/stutters the entire program if Preview is opened.

Note: Although I haven't really messed with large single files in a while...

But when I do, I always do my cleanup/splitting in Calibre first.

Once it's in bite-sized chunks, then I vastly prefer Sigil.

Originally Posted by KevinH

Whoever recommends people to use one large xhtml file is really off the mark and does not understand anything about how technology or browsers really work.

Again, the monolithic HTML file is usually a temporary initial step.

For example, merging together a Gutenberg mess or importing a Word "Save As HTML".

You initially have your huge file, then do your splitting.

But if that initial import is brutal, and Sigil is stuttering/freezing up... it makes it frustrating even if you knew what a subset of the underlying problem was (View > Preview (F10)).

* * *

Artificial Test Case

Take any reasonably large EPUB (>1MB), merge into a single HTML file, then poke away at Code View/Preview.

Here's a great example of a 3.9 MB EPUB (~1.4 million words):

"The Complete Libertarian Forum"

I just tested Sigil 1.5.1 in my VM:

1. Merged all chapters in the EPUB into "Chapter1.html".

2. Clicking View (or anywhere else) instantly froze up Sigil for a while.

3. Once the menu finally opened, even moving the mouse or hovering over menu items caused a huge delay + Sigil (Not Responding). Clicking or doing anything froze it up for another many seconds.

Calibre's Editor handles that type of "monolithic mess" at full-speed.

* * *

Side Note: I also believe this happens when:

• Copying/pasting in huge amounts of material.
• Merging lots of files together + updating all the links

Where Calibre copies/pastes in a second (may have to do with syntax highlighting + previewing in a separate thread?)... Sigil completely chugs.

I also think that EPUB's a brutal case to test merging lots of files together + updating all the internal links.

Here were my rough timings:

Calibre: ~2 seconds to merge all files. Instantly able to edit.

Sigil: ~60 seconds to merge all files. ~45 more seconds until Code View was even scrollable/clickable. (Preview was closed. Didn't retest with Preview open.)

[KevinH]

I am curious ... is the fast time for calibre to merge, is that done with Preview actively showing the file being merged into, or a different file, or is that with the Preview updating message showing?

A second thread just to load Preview is doable but seems like overkill. But waiting for Preview to load could be handled more asynchronously, as long as it does not interfere with syncing from Preview back to CodeView.

I assume calibre is using multiple threads to handle the merging as well. That might make some sense to look into.

[Tex2002ans]

Originally Posted by KevinH

I am curious ... is the fast time for calibre to merge, is that done with Preview actively showing the file being merged into, or a different file, or is that with the Preview updating message showing?

I made sure to keep the "Chapter1.html" tab open as the active tab.

I just tested Calibre again, and it looks like it does this:

1. Preview is generated.

2. Keep "Chapter1.html" open as the active tab.

3. Shift-click + highlight all chapter files.

Right-Click > "Merge selected text files"

Select Chapter1.html bullet.

4. It looks like preview window still keeps the old Chapter1 preview cache (?) displaying there.

5. The instant you click within Code View, the preview window kicks into "Loading preview, please wait..." and takes ~6 seconds to generate a new one.

But as I said, the entire time, you're able to edit code, open up menus, etc. etc. all at full-speed.

Sigil, at that point, every single thing you click on or do became a slog.

Also, as I was typing my post above (~30 mins), I noticed when I clicked back into the merged Sigil window, the entire window re-froze up and went back to "Not Responding" for maybe ~15 seconds. (Preview not open.)

Originally Posted by KevinH

A second thread just to load Preview is doable but seems like overkill. But waiting for Preview to load could be handled more asynchronously, as long as it does not interfere with syncing from Preview back to CodeView.

I assume calibre is using multiple threads to handle the merging as well. That might make some sense to look into.

[KevinH]

One last question since I do not use calibre ... does clicking in their Preview make CodeView sync to the same place and visa-versa like Sigil does? And if so, what happens if you try to sync from Preview to CodeView when Preview is cached during the merge time window, before the updating message comes up?

[KevinH]

Also does calibre properly test all files to be merged to check if well-formed *before* starting the merge and giving you a chance to abort before the merge? Does it check for duplicate or missing url fragments in all files and give you a chance to abort before starting the merge?

Right now Sigil is checking for well-formed before, during, and after which seems to be a bit of overkill since testing just before should be enough. That should make for a simple speedup.

[Tex2002ans]

Originally Posted by KevinH

One last question since I do not use calibre ... does clicking in their Preview make CodeView sync to the same place and visa-versa like Sigil does?

Yep. Just like Sigil.

A click in Preview jumps you to exact location in Code View.

Side Note: Calibre also has 2 helpful buttons below Preview:

• "Disable auto-reload of Preview"
• "Disable syncing of preview position to editor position"

(I have whatever the defaults are... both of those buttons are off.)

Originally Posted by KevinH

And if so, what happens if you try to sync from Preview to CodeView when Preview is cached during the merge time window, before the updating message comes up?

Okay. I just tried it.

When you:

1. Merge

2. Press into Code View.

Preview says "Loading preview, please wait..." for a few seconds. Everything works great and is responsive the entire time.

When you:

1. Merge

2. Press into Preview.

Calibre stuttered a little bit. When I tried to scroll or click around in Preview, it wouldn't work, and neither would Code View (or the entire Calibre window). But within ~6 seconds, it recreated another Preview and everything was back to normal.

Originally Posted by KevinH

Also does calibre properly test all files to be merged to check if well-formed *before* starting the merge and giving you a chance to abort before the merge?

Looks like it just auto-corrects with no acknowledgement.

I went into Chapter2.html and busted one of the tables by removing the closing </tr></table>:

Spoiler:

 <table class="bdr" summary="0" width="100%">
    <colgroup>
      <col width="30%"/>
      <col width="40%"/>
      <col width="30%"/>
    </colgroup>

    <tr valign="middle">
      <td>Joseph R. Peden, Publisher</td>

      <td class="center">Washington Editor, Karl Hess</td>

      <td class="right">Murray N. Rothbard, Editor</td>
    </tr>

    <tr valign="middle">
      <td id="ch_2">VOL. I, NO. II</td>

      <td class="center">APRIL 15, 1969</td>

      <td class="right">35¢</td>
    </tr>          <---- Removed
  </table>         <---- Removed
  
    <h2 class="cts" id="ch_2_1">Tax Day</h2>

[... All of Chapter 2's code...]

and Calibre's merge silently "corrected" + shoved it at the very end of "Chapter2.html"'s code:

Spoiler:

[... All of Chapter 2's code...]

<table class="bdr" summary="0" width="100%">
    <colgroup>
      <col width="30%"/>
      <col width="40%"/>
      <col width="30%"/>
    </colgroup>

    <tbody><tr valign="middle">
      <td>Joseph R. Peden, Publisher</td>

      <td class="center">Washington Editor, Karl Hess</td>

      <td class="right">Murray N. Rothbard, Editor</td>
    </tr>

    <tr valign="middle">
      <td id="ch_2">VOL. I, NO. II</td>

      <td class="center">APRIL 15, 1969</td>

      <td class="right">35¢</td>
 

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

  

</tr></tbody></table>

so in the final merge, you had:

- Chapter1's HTML
- Chapter2's HTML
--- With busted <table> removed from location, placed at end before Chapter3, and corrected.
- Chapter3's HTML

Originally Posted by KevinH

Does it check for duplicate or missing url fragments in all files and give you a chance to abort before starting the merge?

Looks as if it regenerates new ids if needed.

Like I gave it 3 chapters with the same:

<p class="footnote"><a href="#ft.ast1" id="fn.ast1">[*]</a> [...] </p>

and Calibre's merge auto-updated to:

<p class="footnote"><a href="#ft.ast1" id="fn.ast1">[*]</a> [...] </p>
<p class="footnote"><a href="#ft.ast1_1" id="fn.ast1_1">[*]</a> [...] </p>
<p class="footnote"><a href="#ft.ast1_2" id="fn.ast1_2">[*]</a> [...] </p>

I believe it silently does everything, no warnings. (Although I'm no expert in Calibre's innards... I don't usually feed it mangled code. :P)

And on non-existent/duplicate IDs. No, Calibre doesn't warn about those on merge. It only warns about them when you Tools > Check Book (F7) (similar to an epubcheck) or look in the Reports.

Originally Posted by KevinH

Right now Sigil is checking for well-formed before, during, and after which seems to be a bit of overkill since testing just before should be enough. That should make for a simple speedup.

[KevinH]

Okay I have Preview being loaded asynchronously in my local tree which helps but the bulk of the time is spent in merge pre-validating xhtml is well formed and that ids are valid.

If we skip those checks (for the time being only) and let gumbo fix things if needed and use multiple threads to update and extract the pieces to be merged, I can cut down that merge time significantly.

But that is a bruiser of a test case once merged. It has over 40,000 lines of html code.

We synchronously wait for syntax highlighting to be done on purpose to better keep track of actual editing changes vs xhtml syntax colouring changes.

Not sure I want to change that given how rare extreme cases like this are and they should be immediately split so should be short lived issues.

[DiapDealer]

I'm testing a few things that @Tex2002ans described, and I can verify the long periods when Sigil is unresponsive after merging a bunch of files together in the test epub he mentioned above. Many of them I would expect, but the UI interface freezing in general (after the merge is done) seem odd to me (as well as unique unique to Windows, in my experience).

All of my experiments/observations were done with the Preview window closed, with Sigil and it's various files/folders exempt from Windows Antimalware, and with no Power Management restrictions to contend with.

The merge of the 60(ish) chapters itself took well over a minute (i3 8Gb RAM). During that time, processor/ram/disk/gpu usage was not particularly high (both cumulatively--for all processes--and for Sigil specifically). One notable thing is that the Windows 10 task manager indicates that power consumption is "very high" during the merge process. The progress indicator for the merge job immediately jumps to 98% and then Sigil becomes unresponsive for over a minute until the merge is complete.

After the merge is complete, the UI is unresponsive for a shorter period of time, but lengthy nonetheless. To make matters worse, any Window that covers the Sigil window even briefly (including the task manager which frustrates testing) starts the unresponsiveness of Sigil's UI all over again. From thins point on, I can't really say what might trigger another round of ui unresponsiveness (with the exception of Sigil's window being occluded by another app) when trying to open menus or preferences, but when it happens, the length of time is the same.

During the unresponsive periods, Sigil's processor and memory usage spike a bit (but nothing drastic, and not nearly enough to interfere with the running of other applications) and the power consumption indicator immediately spikes to "very high". The gpu % never moves from 0% for Sigil during any of my testing.

With Preview closed and no editing being done, I guess I'm not certain why these lengthy periods of UI/menu unresponsiveness are still happening? I get the merge itself being intensive, but once done, why does Sigil not become more responsive when no editing, or rendering is being done?

Granted... this is all with the monolithic html tab open that I rarely deal with (by choice) myself, but it would seem that Sigil's Windows performance in general might be improved if we could eliminate some (seemingly) unnecessary reparsing,

EDIT: Just to clarify, this is in no way new to Sigil 1.5.1.

[KevinH]

Okay, I have looked into this further. With that testcase merging *all* subsequent chapters into Chapter one.

- a bunch of time verifying all xhtml is wellformed
- a bunch if time verifying no ids are undefined

ignoring those for now, the rest of the time is split up in 4 pieces

- doing the actual merge itself 2 seconds
- deleting all of the resources that was merged about 30 seconds
- updating all other links in non-merged files and the ncx, about a second

The remaining time is spent loading CodeView with the merged file (fast) and then CV running its synchronized syntax highlighting which includes completely spell checking the entire document and then loading Preview (fast since now asynchronous in my local build).

So if you use Preferences to turn off spellchecking in CodeView (no highlighting) you see a huge speedup.

So could someone turn off Preview, and turn off CV spellchecking in Preferences, and try timing that merge testcase again? How much improvement did you get?

[KevinH]

As for the time to delete the now merged resources, it takes a long time because it requires parsing the opf, updating the manifest, spine, guide, and any nav landmarks (so parsing and recreating the nav each time too), then rebuilding the opf and saving it each time repeated for over 100 resources.

Only the merge case would require deleting so many resources at once, so it was designed for single use and safety not speed. I will look into adding a bulk removal routine to save some time.

[DiapDealer]

So could someone turn off Preview, and turn off CV spellchecking in Preferences, and try timing that merge testcase again? How much improvement did you get?

Will do. But before I do...

Do you have any thoughts as to what is being (re)triggered when clicking on a menu item after everything has completely settled down after the merge and the CV reload? It seems like something is being unnecessarily "redone" at that point. All is quiet and done and then clicking any menu item triggers another lengthy freeze. I can't even get the preferences dialog to fully load after the merge is complete. The widget is blank white, the cursor will spin for a minute or so (titlebar says "not responding") then it will "reset" (for lack of a better word), but soon go back to not responding. Lather,rinse, repeat for about 5 minutes now, and still the Preferences dialog has not rendered anything.

EDIT: after about 8 full minutes, the Preferences dialog finally rendered and everything settled down.

I'll try to get some tests done with spellcheck disabled.

[DiapDealer]

Bad news: turns out spellchecking in Code View was already disabled.

The same cycle of the Preferences dialog repeatedly doing something for 8 minutes or so before finally rendering happens every time (at least while the new monolithic merged html tab is the one active) after the merge is complete.

Sigil debug logging is disabled, by the by. I learned my lesson on that when testing Sigil performance issues on Windows