calibre-debug --export-all-calibre-data hangs at metadata.db

[shayaknyc]

Though I said I'd wait, I was curious if I could compile it myself. So, I followed the instructions on this page: https://github.com/kovidgoyal/calibr...ypy/README.rst

I cloned both repos into a clean working directory, entered the calibre directory and issued the bootstrap command, after confirming all the dependencies were installed (as per this: https://github.com/kovidgoyal/calibr...y/sources.json).

I then followed the next step to bootstrap, and this is what happened:

shayaknyc@Garuda ~/../calibre> ./setup.py bootstrap
Already up to date.

*
* Running build
*

Package libpostproc was not found in the pkg-config search path.
Perhaps you should add the directory containing `libpostproc.pc'
to the PKG_CONFIG_PATH environment variable
Package 'libpostproc' not found
Package libpostproc was not found in the pkg-config search path.
Perhaps you should add the directory containing `libpostproc.pc'
to the PKG_CONFIG_PATH environment variable
Package 'libpostproc' not found
Building 31 extensions
Compiling 4 files...
g++ -Wall -DNDEBUG -fno-strict-aliasing -pipe -O3 -fPIC -fvisibility=hidden -pthread -fdiagnostics-color=always -I/usr/include/python3.14 '-DCALIBRE_MODINIT_FUNC=extern "C" __attribute__ ((visibility ("default"))) PyObject*' -std=c++17 -I/usr/include -I/usr/include/onnxruntime -c /home/shayaknyc/git/kovidgoyal/calibre/src/calibre/utils/tts/piper.cpp -o /home/shayaknyc/git/kovidgoyal/calibre/build/objects/piper/piper.o
/home/shayaknyc/git/kovidgoyal/calibre/src/calibre/utils/tts/piper.cpp: In function ‘PyObject* phonemize(PyObject*, PyObject*)’:
/home/shayaknyc/git/kovidgoyal/calibre/src/calibre/utils/tts/piper.cpp:192:20: error: ‘espeak_TextToPhonemesWithTerminator’ was not declared in this scope
  192 |         phonemes = espeak_TextToPhonemesWithTerminator(
      |                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/shayaknyc/git/kovidgoyal/calibre/src/calibre/utils/tts/piper.cpp: In function ‘PyObject* start(PyObject*, PyObject*)’:
/home/shayaknyc/git/kovidgoyal/calibre/src/calibre/utils/tts/piper.cpp:313:32: error: ‘espeak_TextToPhonemesWithTerminator’ was not declared in this scope
  313 |         const char *phonemes = espeak_TextToPhonemesWithTerminator(
      |                                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Can't proceed from here.

[DNSB]

One potential issue with running from source is the current version in the GitHub repository needs the latest beta calibre 8.99.xx (xx is the version, currently 12) to work so you would need to obtain the Linux installer tarball (calibre-8.99.xx-<architecture>.txz where <architecture is either arm64 or x86_64) from the Index of /betas/ link and follow the instructions in the Manual Binary Install in the calibre Download for Linux item.

[kovidgoyal]

Originally Posted by shayaknyc

I guess I'll have to wait, but I am stymied by this, and cannot understand why it's not working. Any ideas as to what would cause the executable not to load from the path specified in the env var?

The most common reason is not setting the env var correctly.

[shayaknyc]

Originally Posted by kovidgoyal

The most common reason is not setting the env var correctly.

I've demonstrated that the env var has been set correctly with the echo command showing that the var has the full path to the src directory, so, not sure how that's possible.

[shayaknyc]

I've updated to the latest version 9.10, and having the same issue with the backup it hangs at metadata.db and doesn't proceed from there.

$ calibre --version
calibre (calibre 9.1)
$ calibre-debug --export-all-calibre-data /home/shayaknyc/ayp/Calibre-Backup-20260202/ all
Exporting libraries: /home/shayaknyc/win-d/Users/shaya/Documents/Calibre Library to: /home/shayaknyc/ayp/Calibre-Backup-20260202/
Failed to initialize plugin: '/home/shayaknyc/.config/calibre/plugins/ANOBII.zip'
Traceback (most recent call last):
  File "calibre/customize/ui.py", line 854, in initialize_plugins
  File "calibre/customize/ui.py", line 88, in load_plugin
  File "calibre/customize/zipplugin.py", line 320, in load
  File "importlib/__init__.py", line 88, in import_module
  File "<frozen importlib._bootstrap>", line 1398, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1371, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1342, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 938, in _load_unlocked
  File "calibre/customize/zipplugin.py", line 212, in exec_module
  File "calibre/customize/zipplugin.py", line 208, in get_code
  File "calibre_plugins.ANOBII.__init__", line 39
        cached_cover_url_is_reliable = True
    ^
TabError: inconsistent use of tabs and spaces in indentation
Failed to initialize plugin: '/home/shayaknyc/.config/calibre/plugins/Barnes & Noble.zip'
Traceback (most recent call last):
  File "calibre/customize/ui.py", line 854, in initialize_plugins
  File "calibre/customize/ui.py", line 88, in load_plugin
  File "calibre/customize/zipplugin.py", line 320, in load
  File "importlib/__init__.py", line 88, in import_module
  File "<frozen importlib._bootstrap>", line 1398, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1371, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1342, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 938, in _load_unlocked
  File "calibre/customize/zipplugin.py", line 219, in exec_module
  File "calibre_plugins.barnes_noble.__init__", line 11, in <module>
    from urllib import quote
ImportError: cannot import name 'quote' from 'urllib' (/opt/calibre/lib/calibre-extensions/python-lib.bypy.frozen/urllib/__init__.pyc)
calibre_plugins.doc_input.__init__:26: SyntaxWarning: "\P" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\\P"? A raw string is also an option.
calibre_plugins.doc_input.__init__:27: SyntaxWarning: "\P" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\\P"? A raw string is also an option.
Failed to initialize plugin: '/home/shayaknyc/.config/calibre/plugins/DOC Input.zip'
Traceback (most recent call last):
  File "calibre/customize/ui.py", line 854, in initialize_plugins
  File "calibre/customize/ui.py", line 88, in load_plugin
  File "calibre/customize/zipplugin.py", line 339, in load
calibre.customize.InvalidPlugin: The plugin at /home/shayaknyc/.config/calibre/plugins/DOC Input.zip cannot be used on linux
Failed to initialize plugin: '/home/shayaknyc/.config/calibre/plugins/KoboBooks.zip'
Traceback (most recent call last):
  File "calibre/customize/ui.py", line 854, in initialize_plugins
  File "calibre/customize/ui.py", line 88, in load_plugin
  File "calibre/customize/zipplugin.py", line 320, in load
  File "importlib/__init__.py", line 88, in import_module
  File "<frozen importlib._bootstrap>", line 1398, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1371, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1342, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 938, in _load_unlocked
  File "calibre/customize/zipplugin.py", line 219, in exec_module
  File "calibre_plugins.kobobooks.__init__", line 10, in <module>
    import time, re, HTMLParser
ModuleNotFoundError: No module named 'HTMLParser'
Failed to initialize plugin: '/home/shayaknyc/.config/calibre/plugins/Read MP3 AudioBook metadata.zip'
Traceback (most recent call last):
  File "calibre/customize/ui.py", line 854, in initialize_plugins
  File "calibre/customize/ui.py", line 88, in load_plugin
  File "calibre/customize/zipplugin.py", line 320, in load
  File "importlib/__init__.py", line 88, in import_module
  File "<frozen importlib._bootstrap>", line 1398, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1371, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1342, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 938, in _load_unlocked
  File "calibre/customize/zipplugin.py", line 219, in exec_module
  File "calibre_plugins.audiobookmetadatareader.__init__", line 8, in <module>
    from common import DEBUG
ModuleNotFoundError: No module named 'common'
/home/shayaknyc/win-d/Users/shaya/Documents/Calibre Library 0 2
metadata.db 0 2824

[kovidgoyal]

Then I need a way to reproduce the issue. First try with all your third party plugins disabled. If it still doesnt work then try exporting only one library and when you fond a library for which it is failing, zip up the library folder and share it with me so I can try to reproduce. Ideally a small library with just some text books.

[kovidgoyal]

Also from the paths in your traceback you have your library shared between windows and linux. Note that this is a recipe for data loss as the two have very different file system semantics. Try moving your library onto a decent linux filesystem (there are very few actually decent ones, ext4 is IMO the least buggy).

[shayaknyc]

Originally Posted by kovidgoyal

Then I need a way to reproduce the issue. First try with all your third party plugins disabled. If it still doesnt work then try exporting only one library and when you fond a library for which it is failing, zip up the library folder and share it with me so I can try to reproduce. Ideally a small library with just some text books.

I've removed the plugins that are causing these errors (I'll have you know, however, that all previous backups proceeded without any issue even with these errors) and still the backup hangs at the metadata.db file. I've made a separate library with just a subset of books, and I'm still getting this error.

calibre-debug --export-all-calibre-data /home/shayaknyc/ayp/Calibre-Backup-20260202 /home/shayaknyc/Documents/Calibre-Subset
Exporting libraries: /home/shayaknyc/Documents/Calibre-Subset to: /home/shayaknyc/ayp/Calibre-Backup-20260202
/home/shayaknyc/Documents/Calibre-Subset 0 2
metadata.db 0 8

What's the best way to share the zipped library folder?
Edit: compressed and attached subset library

Originally Posted by kovidgoyal

Also from the paths in your traceback you have your library shared between windows and linux. Note that this is a recipe for data loss as the two have very different file system semantics. Try moving your library onto a decent linux filesystem (there are very few actually decent ones, ext4 is IMO the least buggy).

Calibre used to be installed on windows, the path is just a convention for me to know that this was data that came from my Windows "D:" drive. The filesystem is ext4.

[kovidgoyal]

Does not reproduce for me:

rm -rf /t/cb; time calibre-debug --export-all-calibre-data /t/cb /t/Calibre-Subset              
Exporting libraries: /t/Calibre-Subset to: /t/cb
/t/Calibre-Subset 0 2
metadata.db 0 7
notes.db 1 7
Pathogenesis: A History of the World in Eight Plagues 2 7
A Sleight of Shadows 3 7
Rarest Blue 4 7
The Gift of Fear 5 7
A Long Walk to Water 6 7
Completed 7 7
Settings and plugins 1 2
Completed 2 2

real	0.487
user	0.461
sys	0.073
maxmem	94 MB
faults	0

[shayaknyc]

Ok, I appreciate that you've managed to get it working. Not sure how I should go about continuing to troubleshoot this, as I rely on the export as a good backup system. Alternatively, I guess I could write a script that backs up my /home/shayaknyc/.config/calibre directory and the library directory itself..... but there elegance of restoring via the backup solution has always been very good.

Any additional ideas you can offer that would help me pinpoint what could be causing this to hang?

[kovidgoyal]

No clue, there is something deeply weird on your system. If you look at your busy error it comes from line 2721 in db/backend.py but that error is explicitly suppressed on line 2720 so it should be impossible.

[shayaknyc]

Originally Posted by kovidgoyal

No clue, there is something deeply weird on your system. If you look at your busy error it comes from line 2721 in db/backend.py but that error is explicitly suppressed on line 2720 so it should be impossible.

Here's a fresh output, I'm seeing this error on line 2742:

$ calibre-debug --export-all-calibre-data /home/shayaknyc/ayp/Calibre-Backup-20260202 /home/shaya/Documents/Calibre-Subset
Exporting libraries: /home/shayaknyc/Documents/Calibre-Subset to: /home/shayaknyc/ayp/Calibre-Backup-20260202
/home/shayaknyc/Documents/Calibre-Subset 0 2
metadata.db 0 7
^CTraceback (most recent call last):
  File "calibre/db/backend.py", line 2742, in _backup_database
apsw.BusyError: not an error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "runpy.py", line 198, in _run_module_as_main
  File "runpy.py", line 88, in _run_code
  File "site.py", line 47, in <module>
  File "site.py", line 43, in main
  File "calibre/debug.py", line 308, in main
  File "calibre/utils/exim.py", line 554, in run_exporter
  File "calibre/utils/exim.py", line 213, in export
  File "calibre/db/cache.py", line 89, in call_func_with_lock
  File "calibre/db/cache.py", line 3354, in export_library
  File "calibre/db/backend.py", line 2747, in backup_database
  File "calibre/db/backend.py", line 2741, in _backup_database
  File "contextlib.py", line 450, in __exit__
KeyboardInterrupt

[kovidgoyal]

Whats that got to do with anything, look at the source corresponding to the version you are running and you will see its the same line, it has just moved because of other changes to the file.

[shayaknyc]

Originally Posted by kovidgoyal

Whats that got to do with anything, look at the source corresponding to the version you are running and you will see its the same line, it has just moved because of other changes to the file.

Ok, well, I cannot seem to get the binary installed in /opt/calibre/ to respect the env var for the dev source, and the latest version's export seems to not work on my machine at all, despite the changes to the source .py file.

I don't have anything unusual in my system, this is an Archlinux based distribution (Garuda, not that it matters). The calibre binary is installed via the linux installer script, not via the pacman. Whether I set the env var in bash or fish (my default shell), the experience is the same. At this point, I'm kind of at a loss for how to proceed. You admitted yourself that the script's next line is meant to suppress the error, yet, it's not working. Would an uninstall and reinstall of the binary help? Should I do a wholesale removal of the binaries and the ~/.config/calibre directory?

Edit: I wanted to test something. I spun up a VM with a minimal ubuntu 24.04 installation, no gui. I installed Calibre via the linux installer script, and I copied the Calibre-Subset library over to the VM and attempted to perform a backup. This is the output:

shaya@calibre-ubuntu:~$ calibre-debug --export-all-calibre-data /home/shaya/Calibre-Backup-20260202 /home/shaya/Calibre-Subset
Traceback (most recent call last):
File "runpy.py", line 198, in _run_module_as_main
File "runpy.py", line 88, in _run_code
File "site.py", line 47, in <module>
File "site.py", line 43, in main
File "calibre/debug.py", line 308, in main
File "calibre/utils/exim.py", line 552, in run_exporter
KeyError: '/home/shaya/Calibre-Subset'

I can confirm that calibre can read the library, see here:

shaya@calibre-ubuntu:~$ calibredb list --with-library /home/shaya/Calibre-Subset/
id title                                                authors
1 Pathogenesis: A History of the World in Eight        Jonathan Kennedy
Plagues
2 A Sleight of Shadows                                 Kat Howard
3 Rarest Blue                                          Baruch Sterman & Judy Taubes Sterman
4 The Gift of Fear                                     Gavin de Becker
5 A Long Walk to Water                                 Linda Sue Park

While I appreciate that you're unable to reproduce the error, I'm not sure why on a fresh VM, with a fresh install of Calibre, that this would happen?

[kovidgoyal]

That's a different error. You need to open the libraryin the calibre program so that calibre knows about it. Once you do so you wont have that error any more.