calibre-debug --export-all-calibre-data hangs at metadata.db

[kovidgoyal]

The only way that error can happen is if you have some other process modifying metadata.db while the backup is running.

[shayaknyc]

At long last, after nearly a year, I was finally able to pinpoint the actual issue!

Cause: the export destination was on an SMB/CIFS mount, and SQLite cannot take a file lock there.

Since 8.13.0 the export creates its temporary copy of metadata.db inside the export destination instead of the system temp directory. That came in with this commit, for bug 2126863:
https://github.com/kovidgoyal/calibr...996021995a08b8

My destination has always been a NAS share mounted over CIFS. Up to 8.12.0 that was fine, because the temporary database lived in /tmp and only the finished part files were written to the NAS. From 8.13.0 on, SQLite has to open and lock a database on the share itself, every lock attempt comes back busy, and _backup_database() suppresses BusyError and retries forever. That is the hang at "metadata.db 0 N". My last good backup was on 4 Oct 2025, the day before that commit.

It also explains why the issue could not be reproduced with my subset library: the test exported to a local directory. Every one of my failing runs, including the subset and the Docker container, wrote to the same NAS mount. I should have mentioned where the destination lived; it never occurred to me that it mattered. The library, its filesystem, the plugins and the custom columns were all irrelevant.

To reproduce (calibre 9.15, Linux, any library):

# destination on local disk: completes in seconds
calibre-debug --export-all-calibre-data /tmp/export-test/ /path/to/library

# destination on a CIFS mount: hangs at metadata.db
calibre-debug --export-all-calibre-data /mnt/cifs-share/export-test/ /path/to/library

While it hangs there is a zero byte tmpXXXX-metadata.db in the destination. Plain SQLite fails the same way on that mount, with no calibre involved:

python3 -c "import sqlite3; c=sqlite3.connect('/mnt/cifs-share/t.db', timeout=5); c.execute('create table t(x)')"
sqlite3.OperationalError: database is locked

The share was mounted with vers=3.0 and otherwise default locking options.

What fixed it for me: mounting the same NAS folder over NFS (v4) instead of CIFS. SQLite locks work there and the export is running normally again. Exporting to a local disk and copying the result to the NAS afterwards would also work.

A suggestion, if you think it is worthwhile: the difficult part of this was that the failure is silent. Because BusyError is suppressed inside the loop, an unlockable destination looks exactly like a hang, with no message at all. Giving up after a bounded number of busy retries with an error along the lines of "cannot lock the temporary database in the export destination" would have pointed straight at it. Falling back to the system temp directory when the destination cannot be locked would avoid the problem altogether.

Thank you for sticking with this thread for as long as you did, and to everyone else who chimed in.

[kovidgoyal]

https://github.com/kovidgoyal/calibr...f5a631051f4436

[shayaknyc]

Originally Posted by kovidgoyal

https://github.com/kovidgoyal/calibr...f5a631051f4436

Awesome!