Sigil 2.8.6 crashes on Windows11 when adding a cover to a large EPUB3

[rinsay]

Sigil 2.8.6 crashes on Windows when I use Add Cover on a large EPUB3 with
no existing cover. I can reproduce it by opening the EPUB, choosing Add
Cover, and selecting an image. The crash occurs even when I skip Mend
and Prettify.

I'm using Sigil 2.8.6 on Windows x64, build 26100. The book was exported
by txt2epub, an open-source browser-based converter.
Source: https://github.com/RinSayX/txt2epub
Live instance used for export: https://rinsay.com

I tested the same cover image in each case:

- Blank EPUB3 created in Sigil: works.
- Small demo novel exported by txt2epub: works.
- First 2 chapters of the affected novel: works.
- First 1,200 chapters of the same novel: crashes.
- Same 1,200-chapter EPUB with the TOC in nav.xhtml reduced to one entry,
while keeping the chapter files and reading order: still crashes.
- Full novel with 2,226 reading-order entries: crashes.

The original full EPUB passed EPUBCheck 5.4.0 with zero errors and zero
warnings. I was able to add the same cover and save the book in Sigil
2.8.1 on macOS.

Windows Event Viewer and WinDbg showed these failures on separate runs:

- Event Viewer: ntdll.dll, exception 0xc0000374 (heap corruption).
- WinDbg capture 1: 0xc0000409, FAST_FAIL_STACK_COOKIE_CHECK_FAILURE.
The stack includes sigilgumbo!gumbo_string_buffer_append_codepoint
calling ucrtbase!realloc_base.
- WinDbg capture 2, after reducing the TOC to one entry: 0xc0000005,
attempting to read address 0x8 in python314!PyEval_EvalFrameDefault.

I have the full WinDbg stack traces. Has anyone seen similar behavior
when adding a cover to a book with many XHTML files? Is there a debug
build or additional logging that would help narrow down the cause?

Thanks!

[KevinH]

Your debug info shows the bug has heap corruption when sigil gumbo tries to reallocate memory on the stack. My guess is the size of the epub is using up the available stack on one thread but I am not sure why that would cause heap corruption instead of code or stack corruption.

My guess is your Windows box does not have enough memory ... or maybe a recursion (which is capped in Sigil) during parsing is happening that is resulting in a stack limit being reached. On macOs or unix you can use ulimit to set appropriate runtime stack sizes. I am not sure how to control the default C++ stack size on Windows. You could also be reaching a limit on the number of open file descriptors. These is nothing specific to the Sigil codebase that should be limiting you.


Please zip up your epub, then post it someplace, and PM a web link to KevinH (me) here on Mobileread. I will use it to try to recreate what you are seeing on Linux and MacOS.

Having over 2000 chapters/files in a single epub does seem like a bit extreme. Is there some reason for so many files?

[DiapDealer]

I'd love to confirm that the same thing happens using the same epub with Sigil 2.8.1 and earlier on Windows.

[DiapDealer]

Would one of our embedded Python routines even be involved in adding a cover image?

- WinDbg capture 2, after reducing the TOC to one entry: 0xc0000005,
attempting to read address 0x8 in python314!PyEval_EvalFrameDefault.

Normally I would suspect a flawed C extension with that kind of Python error. But regardless, that error indicates segfault/crash of the Python interpreter itself.

[KevinH]

Yes python is used by the lxml python pure xml parser. So adding and removing metadata and manifest properties for the cover would need to use lxml to parse the OPF for a sanity check both before reading and after writing for sanity.

My guess is this is not a general Sigil 2.8.6 issue but a Windows machine issue with limited stack space, limited memory, or limited file descriptors. For speed, gumbo can be invoked by QtConcurrent with multiple chapters being parsed at the same time. Ths can eat up open file descriptors like crazy.

[KevinH]

I have one testcase with like 4000 images so I will try with that. I do not think I have any epubs with over 1000 chapters to test with though. Hope the OP provides his epub as a test case.

[KevinH]

Okay I found one test epub with over 15000 images and one test epub with over 4000 images and adding a cover works just fine (slow at times but just fine) on MacOS.

I do not have even one test epub with over 1000 chapters.

[KevinH]

Okay, I found one BeckyEbook once posted. It has 2500 files in it. It is epub2, but I was able to convert it to epub3. Then it does not have any images to make a cover with. So I added one and I have attached that test epub here:

[KevinH]

Now, when I try to add a cover to this epub3, it happens quite quickly but I noticed that the opf spine was in a random order!

So something is funny, even on MacOS.

Not sure what is going on.

Update:
That test epub was missing its spine, so no order was provided!

[DiapDealer]

Whatever it is, it's not new. I can duplicate the crazy reordering on Windows with Sigil 2.8.1, 2.8.5 and 2.8.6. No crash with Becky's 2500files epub using any of those, though.

EDIT: I've gone back as far as Sigil 2.6.2 and I'm able to add the cover image (on Windows) to the 2500files epub with no crash, but the spine jumbling still happens. This is clearly an old bug that affects epubs with massive amounts of chapters.

[DiapDealer]

I've reduced the number of chapters in the 2500files epub in stages down to 99 chapters, and Add Cover is still jumbling the spine order with Sigil 2.6.2 on Windows. But still no crash.

EDIT: Ok. I'm down to 25 chapters and Add Cover still jumbles the epub3 spine! (using Sigil 2.6.2 and 2.8.6 on Windows). Clearly no one is using Add Cover on epub3s. Not on Windows at least.

[KevinH]

My bad, that test epub was missing almost all of its spine. That was the point of that test epub!

I have fixed that test case and have reposted it above. It now has the spine order specified. Look for the "_fixed" in the name.

Now everything works even with 2500 files and no random order!

So I can not recreate what they reported at all.

Now if you were doing epub2, then this is going to be alot slower as the guide values need to be looked up to find any existing semantics. I can probably speed that up in a future release but it is not a bug and it works just fine (but slow!).

[KevinH]

So there is no general bug with Adding a Cover to an epub with over 1000 chapter files in it.

There might be speed improvements but I am back to thinking the OP's machine is short on memory or allocated stack per thread or ... something.

[DNSB]

I went with the full download and added a cover image. Before the cover was added, there was one item in the <spine> listing. After the image file was added and a xhtml file called cover.xhtml was added, the file listing in the <spine> section of the content.opf was indeed scrambled after all the files were listed which has happened before but since it's part of what the reader sees, it didn't bother me.

  <spine>
    <itemref idref="text0001.xhtml"/>
  </spine>

After I added the cover page, under the <manifest>, there were 2 added lines:

    <item id="cover.jpg" href="Images/cover.jpg" media-type="image/jpeg" properties="cover-image"/>
    <item id="cover.xhtml" href="t/cover.xhtml" media-type="application/xhtml+xml" properties="svg"/>

And the <spine> section now had all the files in a random order.

Saving the file took about 15 seconds which is longer than expected.

Edit: tried the version with the spine fixed and no issues were seen. The save is still on the slow side but likely due to zipping a large number of files.

[KevinH]

If you go regrab the test epub I posted (look for _fixed) in the name, the spine is now correct and the order is kept when adding a cover.