Reports crashes in 0.9.10

[ghmerrill]

This has happened suddenly. My most recent activity (other than some minor changes to some content) was a number of changes in my CSS file. But this passes the CSS tests.

I click on Reports, see the progress bar start to run in its dialog, and then a dialog pops up that says "Sigil has encountered a problem. Sigil may need to close." Show Details reveals:

Error info: Unknown exception
Sigil version: 0.9.10
Runtime Qt: 5.6.2
Compiled Qt: 5.6.2
Platform: Windows Sysinfo ID 192

This also happens on 0.9.9.

I'm running Win 10 Professional (64 bit).

[DiapDealer]

Try to get a small, non-copyrighted* epub sample that crashes consistently which you can share.



(NOTE: Free <> non-copyrighted. Mobileread has strict rules against uploading copyrighted material)

[ghmerrill]

Originally Posted by DiapDealer

Try to get a small, non-copyrighted* epub sample that crashes consistently which you can share.

I was afraid you were going to say this. The current epub this is happening in is a book I'm working on -- with 13 files (currently 8 chapter files, coverpage, TOC, etc.) of about 324K size and approximately 42K words. The book is not copyrighted currently, and I'm the sole author.

I'm willing to put some time into helping debug this and producing a small test case -- if one exists. But my situation is one where I don't want to stop book development and devote excessive time to that exercise. I'm sure you understand. The report function is very convenient for me (largely because it gives me a word count), but it's not essential. Unless, of course, this turns out to be a symptom of a more fundamental error, either in the book files themselves (undiagnosed other than by Reports crashing) or in some part of Sigil code outside of Reports. It does look like something's throwing an exception and it's just not being caught at an informative place.

I'll put a day into attempting to isolate the problem. (I've got about 30 years of software design/development behind me, and if there's a simple test case to be found, I should be able to find it in that time.) I also have a slightly earlier version that doesn't exhibit the problem, but I suspect that's of little value.

If there's any way for me to run this under a debugger, I'd be happy to do that. But I'm really no longer set up to do development myself. Otherwise, stand by, and I should be able to get you a test case by tomorrow with any luck.

[DiapDealer]

Originally Posted by ghmerrill

But my situation is one where I don't want to stop book development and devote excessive time to that exercise. I'm sure you understand.

I understand perfectly.

Originally Posted by ghmerrill

The report function is very convenient for me (largely because it gives me a word count), but it's not essential. Unless, of course, this turns out to be a symptom of a more fundamental error, either in the book files themselves (undiagnosed other than by Reports crashing) or in some part of Sigil code outside of Reports. It does look like something's throwing an exception and it's just not being caught at an informative place.

There's been a lot of "crashy" stuff reported/fixed with the Reports feature lately, but it's usually been the result of one of the underlying documents being altered while the Reports dialog is still open (and subsequently refreshed).

Originally Posted by ghmerrill

I'll put a day into attempting to isolate the problem. (I've got about 30 years of software design/development behind me, and if there's a simple test case to be found, I should be able to find it in that time.) I also have a slightly earlier version that doesn't exhibit the problem, but I suspect that's of little value.

Anything you can provide would be helpful. And if it takes longer than a day, then it takes longer.

[ghmerrill]

It's not associated with your "Reports dialog open" bugs. That's some good news.

The other good news is that it's largely my fault -- partly for trying to be too clever within certain confines of CSS/HTML/XML.

It turns out that one of my HTML files in the book is something I've been using as a "template" for creating a Chapter file for the book. I recently cleaned up and improved all the CSS (yuck!) for that since I use it to auto-generate chapter and section numbering -- but that's largely irrelevant. Except that when I did that I added a highly visible line in a <style> section in that HTML "template" file -- intentionally ill-formed CSS -- to remind me (if I missed it) that I had to set the chapter counter at that point in the HTML file. Sigil, in general, doesn't seem to care about this much, and just in Book View displays the red "This page contains the following errors" box -- which was precisely my goal to avoid overlooking setting that counter.

But apparently this drives reports CRAZY!. It throws an exception (I guess, who can blame it).

So I think that I can say I found the problem and it is I. I guess you might want to catch that exception a little better. Otherwise, Sigil just crashes if you have an error in one of your content files and run Reports on it. You can't really ignore it in Reports since it's an error in the HTML file. But probably not a real high priority.

[DiapDealer]

Yes, thanks. We will certainly want to fix the crash regardless.

But with regard to malformed xhtml, the Reports feature is not the only Sigil feature that could be adversely affected by intentionally leaving things not-well-formed (though they may warn rather than ungracefully crashing).

Can you share an example snippet of a style section in an xhtml file that will make Reports barf on its own shoes? Or are you saying that any invalid style section causes the crash?
Never mind. I've tested myself, that nearly any non-well-formed xhtml will cause a crash when Tools->Reports is initiated.

[ghmerrill]

That's what I'd expect.

[KevinH]

That should not be the case. If gumbo detects not well formed xhtml, the well formed checks should prevent Reports from being launched at all. If you launch Reports and then create broken xhtml and try to refresh it may crash.

[ghmerrill]

A possibly useful feature would be to flag each file in the Book Browser with some indicator that there is an error in it. That way the user wouldn't have to depend on actually looking at the file directly to see that there's a problem with it. But I know that this is just "one more feature" to add.

And in all honesty, I WAS intentionally ignoring the error in the file. Perhaps if I stash it in the Misc folder rather than the Text folder, things will work better. One pays for being sloppy.

[DiapDealer]

Originally Posted by KevinH

That should not be the case. If gumbo detects not well formed xhtml, the well formed checks should prevent Reports from being launched at all. If you launch Reports and then create broken xhtml and try to refresh it may crash.

That's sort of what I thought, too. But I just opened a brand-new empty epub in Sigil, added an opening (and unclosed) p tag, and Sigil crashed as soon as I run Reports (for the very first time).

I'm using a custom version of Sigil that's more recent that 0.9.10, but it sounds like the OP had been experiencing it in 0.9.9, as well. At least on Windows 10 (which is where I just tested it), anyway. I'll test on Linux when I get home (which often mirrors Windows crashes).

[KevinH]

I will try the same thing in my Mac when I get home.

[KevinH]

It crashes for me too. That said if I try the same test with a div tag (a tag that html5 parsing rules do not allow to be unclosed) instead of a p tag, the non-well formed code is properly detected.

So the problem is gumbo is really a self-recovering html5 parser (like browsers use) and not a strict xml parser but the Reports tool uses Qt's overly strict pure xml parser. So according to gumbo an unclosed p tag is okay because the gumbo parser will "fix" it and technically is allowed in html5 according to those parsing rules.

The well-formed check F7 properly detects the unclosed p tag but it is python-based and therefore slower than gumbo which can be used in a multithreaded manner safely.

So either I need to teach gumbo to be more picky and throw more "errors", or we need to rewrite the Reports code to use the gumbo parser instead of the overly strict Qt xml parser.

I think the way forward is to use the gumbo parser to generate the Reports and let Mend handle fixing things when the user wants which should make Reports much less crash prone in general.

What do you think?

[KevinH]

Alternatively, we could replace the more lenient gumbo well-formed check with a check using the QXmlStreamReader itself and catch exceptions and only allow strickly valid xhtml to be used by Reports.

...

[DiapDealer]

Regardless of the method (to which I would defer to your expertise), I would personally prefer the outcome you described in the latter post: namely the Reports feature only running when all xhtml is strictly valid. That would (or should) avoid both the crashing and the potential for the modification of underlying code merely to be able to run the reports.

Have said that, however: I'm not one who really uses the Reports feature. We should probably get the opinions of some who use it regularly. Massage invalid code (invalid by strict xhtml rules) to allow Reports to run without crashing, or not allow Reports to be run when there's strictly invalid xhtml present. Assuming I've interpreted the gist of your propositions accurately, of course.

[KevinH]

After looking, I think we should add a method to the HtmlResource object so that it will use the QXMLStreamReader itself to check itself for xml well-formedness and we can invoke those checks before launching the Reports. That will be easier than either trying to teach gumbo to be less lenient or trying to rewrite all of the Reports pieces to use gumbo.

I will try to get something working and pushed by tomorrow evening.