ebook-convert epub to pdf RuntimeError

[xiatian]

I'm using the lastest linux binary,

ebook-convert b.epub b.pdf
1% Converting input to HTML...
Note 'webEngineArgs' are overridden by QTWEBENGINE_CHROMIUM_FLAGS
QRhiGles2: Failed to create temporary context
QRhiGles2: Failed to create context
This plugin does not support createPlatformVulkanInstance
QVulkanInstance: Failed to initialize Vulkan
Unable to detect GPU vendor.
InputFormatPlugin: EPUB Input running
on /root/test/b.epub
Found HTML cover titlepage.xhtml
Parsing all content...
34% Running transforms on e-book...
Merging user specified metadata...
Detecting structure...
Flattening CSS and remapping font sizes...
Source base font size is 12.00000pt
Removing fake margins...
Cleaning up manifest...
Trimming unused files from manifest...
Creating PDF Output...
67% Running PDF Output plugin
Traceback (most recent call last):
File "runpy.py", line 203, in _run_module_as_main
File "runpy.py", line 88, in _run_code
File "site.py", line 52, in <module>
File "site.py", line 48, in main
File "calibre/ebooks/conversion/cli.py", line 427, in main
File "calibre/ebooks/conversion/plumber.py", line 1265, in run
File "calibre/ebooks/conversion/plugins/pdf_output.py", line 190, in convert
File "calibre/ebooks/conversion/plugins/pdf_output.py", line 239, in convert_text
File "<frozen importlib._bootstrap>", line 1371, in _find_and_load
File "<frozen importlib._bootstrap>", line 1342, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 938, in _load_unlocked
File "bypy-importer.py", line 279, in exec_module
File "calibre/ebooks/pdf/html_writer.py", line 15, in <module>
File "<frozen importlib._bootstrap>", line 1371, in _find_and_load
File "<frozen importlib._bootstrap>", line 1342, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 938, in _load_unlocked
File "bypy-importer.py", line 279, in exec_module
File "html5_parser/__init__.py", line 31, in <module>
RuntimeError: html5-parser and lxml are using different versions of libxml2. This happens commonly when using pip installed versions of lxml. Use pip install --no-binary lxml lxml instead. libxml2 versions: html5-parser: (2, 9, 14) != lxml: (2, 15, 3)

However, 8.x version works normally.

[kovidgoyal]

Dont see how thats possible if you are using the official binaries. Since they come with their own bundled libxml. Use LD_DEBUG=libs to figure out where the spurious libxml is coming from.

[xiatian]

LD_DEBUG=libs ebook-convert b.epub b.pdf 2>&1 | grep -i "libxml2" > /tmp/ld_debug.log
cat /tmp/ld_debug.log
190956: find library=libxml2.so.16 [0]; searching
190956: trying file=/root/calibre/lib/libxml2.so.16
190956: calling init: /root/calibre/lib/libxml2.so.16
190961: find library=libxml2.so.16 [0]; searching
190961: trying file=/root/calibre/lib/libxml2.so.16
190960: find library=libxml2.so.16 [0]; searching
190960: trying file=/root/calibre/lib/libxml2.so.16
190960: calling init: /root/calibre/lib/libxml2.so.16
190961: calling init: /root/calibre/lib/libxml2.so.16
190956: find library=libxml2.so.2 [0]; searching
190956: trying file=/root/calibre/lib/libxml2.so.2
190956: trying file=/lib/x86_64-linux-gnu/libxml2.so.2
190956: calling init: /lib/x86_64-linux-gnu/libxml2.so.2
RuntimeError: html5-parser and lxml are using different versions of libxml2. This happens commonly when using pip installed versions of lxml. Use pip install --no-binary lxml lxml instead. libxml2 versions: html5-parser: (2, 9, 14) != lxml: (2, 15, 3)
190956: calling fini: /lib/x86_64-linux-gnu/libxml2.so.2 [0]
190956: calling fini: /root/calibre/lib/libxml2.so.16 [0]

[kovidgoyal]

https://www.mobileread.com/forums/sh...d.php?t=370085

[xiatian]

Originally Posted by kovidgoyal

https://www.mobileread.com/forums/sh...d.php?t=370085

This is what AI helped me and I think there maybe a fix for the next release.

Thanks for the pointer. Traced this further and it doesn't look like
a plugin issue — here's what I found:

1. calibre-customize -l shows only built-in plugins, no third-party
plugins installed at all.

2. Scanned every .so file under /root/calibre for a compiled ELF
dependency on libxml2.so.2:

find /root/calibre -type f -name "*.so*" | while read f; do
readelf -d "$f" | grep -q "libxml2.so.2" && echo "$f"
done

-> no output. Also did a strings scan across every file in
/root/calibre for the literal string "libxml2.so.2" -> also
nothing found. So nothing is linking against it at the ELF level,
confirming what you said.

3. However, running this with calibre's own bundled python:

/root/calibre/calibre-debug -c "import ctypes.util; print(ctypes.util.find_library('xml2'))"

returns: libxml2.so.2 (the system one, not calibre's bundled
libxml2.so.16)

So it looks like html5-parser is locating libxml2 at runtime via
ctypes.util.find_library() / dlopen rather than a compiled ELF NEEDED
dependency — which is why it doesn't show up in readelf or strings.
find_library() resolves through the system's ldconfig cache, which
only knows about whatever libxml2 the OS has registered — in this
case the system's own libxml2.so.2. It then tries to dlopen that
exact filename; the dynamic linker checks calibre's own lib dir first
(as seen in the LD_DEBUG trace) but finds no file by that name there
(calibre ships libxml2.so.16 under the new SONAME), so it falls
through to the system's copy instead. lxml, by contrast, loads
libxml2.so.16 directly and correctly picks up calibre's bundled copy.
End result: two different libxml2 versions loaded in the same
process.

I also tested this on a second, completely unrelated Debian 12 server
with a different software stack, and got the exact same error. So
it's not tied to any specific package pulling in libxml2 — the common
factor is simply that it's a Debian 12 host, and Debian 12's repo
still ships libxml2 2.9.14 under the old SONAME (libxml2.so.2), while
calibre's own bundled libxml2 has moved to the new SONAME (.so.16).
Given how many common packages transitively depend on libxml2, most
Debian 12 installs probably have some libxml2.so.2 registered in
ldconfig one way or another, so any Debian 12 host running the
official calibre binary seems likely to hit this.

As for a possible fix, I can see two independent ways to close this
gap (either one alone would probably be enough):

1. On the html5-parser side: instead of relying on
ctypes.util.find_library('xml2') (which resolves through the
system's ldconfig cache and has no awareness of calibre's own
bundled lib directory), it could resolve the libxml2 path the same
way lxml does — e.g. reuse the same shared library lxml already
loaded, or at least restrict the search to calibre's own lib dir
before falling back to a system-wide lookup.

2. On the calibre packaging side: since the bundled libxml2 SONAME
changed from .so.2 to .so.16 in this build, shipping a compat
symlink alongside it (libxml2.so.2 -> libxml2.so.16, inside
calibre's own lib dir) would make html5-parser's existing
find_library()-based lookup resolve correctly without any code
change.

For now I've applied the workaround you suggested:

ln -s libxml2.so.16 /root/calibre/lib/libxml2.so.2

which resolves it fine. Just wanted to share the fuller trace in case
it's useful.

[kovidgoyal]

AI has no clue what it's talking about.

ldd html_parser.cpython-314-x86_64-linux-gnu.so| grep xml            
	libxml2.so.16 => /usr/lib/libxml2.so.16 (0x00007fd29e127000)

html5-parser does not use load library.

And I tried, once again, to reproduce this issue in a fresh debian:stable container which has libxml.so.2 installed and it does not reproduce.

/opt/calibre/ebook-convert test.txt .pdf

works as expected.

[xiatian]

But it's confusing why 8.x version works.

LD_DEBUG=libs ebook-convert b.epub b.pdf 2>&1 | grep -i "libxml2" > /tmp/ld_debug.log
cat /tmp/ld_debug.log
3246671: find library=libxml2.so.16 [0]; searching
3246671: trying file=/root/calibre/lib/libxml2.so.16
3246671: calling init: /root/calibre/lib/libxml2.so.16
3246678: find library=libxml2.so.16 [0]; searching
3246678: trying file=/root/calibre/lib/libxml2.so.16
3246679: find library=libxml2.so.16 [0]; searching
3246679: trying file=/root/calibre/lib/libxml2.so.16
3246679: calling init: /root/calibre/lib/libxml2.so.16
3246678: calling init: /root/calibre/lib/libxml2.so.16
3246671: calling fini: /root/calibre/lib/libxml2.so.16 [0]

[xiatian]

After a lot of testing, I think I've found the actual culprit — it's
QtWebEngine's GPU probing on GPU-less servers, not html5-parser
itself.

The workaround that fixed it:

QTWEBENGINE_CHROMIUM_FLAGS="--no-sandbox --disable-gpu --disable-software-rasterizer" ebook-convert b.epub b.pdf

The full investigation with AI support is as follows:

Finally nailed it down completely. It's not an html5-parser bug at
all — it's QtWebEngine's GPU/EGL probing on GPU-less servers.

Root cause, confirmed via full LD_DEBUG traces comparing calibre 8.13
(works) vs 9.11 (fails) on identical GPU-less Debian 12 hosts:

8.13 — after EGL/GBM init, goes straight into QtWebEngineCore, only
ever loads one libxml2 (calibre's bundled libxml2.so.16). Clean run.

9.11 — after libEGL_mesa.so.0, goes on to actually load Mesa's
software rasterizer fallback:

calling init: libEGL_mesa.so.0
find library=libxml2.so.2 [0]; searching
trying file=/lib/x86_64-linux-gnu/libxml2.so.2
calling init: libicudata.so.72 / libpciaccess.so.0 / libxml2.so.2
calling init: libLLVM-15.so.1
calling init: /usr/lib/x86_64-linux-gnu/dri/swrast_dri.so
...
calling init: html5_parser.html_parser.so
RuntimeError: html5-parser and lxml are using different versions
of libxml2 ... html5-parser: (2, 9, 14) != lxml: (2, 15, 3)

swrast_dri.so is Mesa's software rasterizer (a Debian system package,
not part of calibre), and it pulls in the system's own libxml2.so.2
as a transitive dependency — loading it into the same process as
calibre's bundled libxml2.so.16. Two different libxml2 copies end up
coexisting in one process, and html5-parser's version check reads the
wrong one.

Both servers are equally GPU-less, so 9.11's bundled Qt/QtWebEngine
apparently probes one step further into EGL/GPU detection than 8.13's
did — actually reaching the Mesa software-rasterizer fallback instead
of stopping earlier. Looks like a Qt/QtWebEngine behavior change
between the two calibre builds.

Given that this will affect anyone running 9.11 on a GPU-less
VPS/server (a pretty common calibre-server setup), it might be worth
flagging in the docs, or even defaulting to --disable-gpu when no
display/GPU is detected.

[kovidgoyal]

It will only affect people running on debian I dont think anybody else still ships the highly broken/insecure libxml.so.2 and I still cannot replicate it in debian:stable container.

Not to mention that calibre anyway disables webengine GPU in ebook-convert as it is headless, see line 1853 in gui2/__init__.py

[xiatian]

I'm using Debian 12 and will keep using it for 2-3 years. The --disable-gpu workaround should be my fix.

[JSWolf]

Which version of calibre are you running?