calibre-debug --export-all-calibre-data hangs at metadata.db

[shayaknyc]

Originally Posted by kovidgoyal

That's a different error. You need to open the libraryin the calibre program so that calibre knows about it. Once you do so you wont have that error any more.

Ok, so I loaded Calibre GUI in the VM and pointed it to the Calibre-Subset library (see attached screenshot)

However, after it loads the library correctly, I run the same export command and this is what I got:

$ calibre-debug --export-all-calibre-data /home/shayaknyc/Calibre-Backup-20260202 /home/shayaknyc/Calibre-Subset
Traceback (most recent call last):
  File "runpy.py", line 198, in _run_module_as_main
  File "runpy.py", line 88, in _run_code
  File "site.py", line 47, in <module>
  File "site.py", line 43, in main
  File "calibre/debug.py", line 308, in main
  File "calibre/utils/exim.py", line 552, in run_exporter
KeyError: '/home/shaya/Calibre-Subset'
$

[kovidgoyal]

I dont see how that's possible unless your GUI and CLI commands are accessing different config directories. In any case I have already removed the constraint that export has to work with known libraries for the next release.

[shayaknyc]

Originally Posted by kovidgoyal

I dont see how that's possible unless your GUI and CLI commands are accessing different config directories. In any case I have already removed the constraint that export has to work with known libraries for the next release.

I mean, this doesn't make any sense. Fresh Ubuntu VM, fresh calibre install, same library I shared with you, which you did not seem to have a problem with. So, I'm not at all sure what's causing this behavior....

[shayaknyc]

Just updated to v9.2.1 and same deal. Just hangs at metadata.db.

At this point, I give up. A little disappointing since I scripted the backup and had it activate on a cron.

For clarity, so I can make sure I can continue to backup properly, so long as I continue to back up the ~/.config/calibre directory along with the library directory, I could "restore" manually?

[JSWolf]

Originally Posted by shayaknyc

Just updated to v9.2.1 and same deal. Just hangs at metadata.db.

At this point, I give up. A little disappointing since I scripted the backup and had it activate on a cron.

For clarity, so I can make sure I can continue to backup properly, so long as I continue to back up the ~/.config/calibre directory along with the library directory, I could "restore" manually?

Turn off the cron job. You don't need it Use the Backup plugin for calibre to backup your configuration directory when you exit calibre.

[shayaknyc]

Originally Posted by JSWolf

Turn off the cron job. You don't need it Use the Backup plugin for calibre to backup your configuration directory when you exit calibre.

Where can I find this Backup Plugin?

[JSWolf]

Originally Posted by shayaknyc

Where can I find this Backup Plugin?

In Preferences > Plugins > Get new plugins, in the search box, search for backup and the plugin is called Backup configuration folder(s).

[shayaknyc]

Originally Posted by JSWolf

In Preferences > Plugins > Get new plugins, in the search box, search for backup and the plugin is called Backup configuration folder(s).

Ah, I see it. This is only a partial backup solution however. The command I was using backed up the config AND library in a multi-file archive which was super easy to restore via GUI.

Backing up the two folders I specified on a schedule, essentially does the same thing but with a manual restoration process.

[JSWolf]

Originally Posted by shayaknyc

Ah, I see it. This is only a partial backup solution however. The command I was using backed up the config AND library in a multi-file archive which was super easy to restore via GUI.

Backing up the two folders I specified on a schedule, essentially does the same thing but with a manual restoration process.

The plugin is only for backing up the configuration. However, your way has one flaw. when you have a cron job scheduled, you cannot be using calibre at that time.

[shayaknyc]

Originally Posted by JSWolf

The plugin is only for backing up the configuration. However, your way has one flaw. when you have a cron job scheduled, you cannot be using calibre at that time.

The script checks for any file-locks or if calibre is running and stops and sends a notification to let me know it was unable to complete.

I appreciate your help!

[shayaknyc]

I think I have some clarity on this issue, as it still persists. I decided to try running my backup script inside a docker container running calibre from lscr.io/linuxserver/calibre:latest. The calibre-debug executable hangs at exactly the same place - the metadata.db file. Mine is 15.5mb large, and does have a number of custom columns/fields - would this perhaps be the culprit for why it just hangs at that stage? I can confirm that the Calibre GUI is not running when I execute the backup command inside the container.

Any insights would be very helpful! Thank you!

Edit: confirming that the Calibre Library directory is on an ext4 partition. It consistently fails at the metadata.db file. Wonder if there's a way to run a repair?

[kovidgoyal]

No size is irrelevant. As for repairing, you can always dump it to sqlite and rebuild from that, or use calibredb restore_database to rebuild it completely from OPF files, but read the help of that command to see its limitations.

[JSWolf]

Originally Posted by shayaknyc

I think I have some clarity on this issue, as it still persists. I decided to try running my backup script inside a docker container running calibre from lscr.io/linuxserver/calibre:latest. The calibre-debug executable hangs at exactly the same place - the metadata.db file. Mine is 15.5mb large, and does have a number of custom columns/fields - would this perhaps be the culprit for why it just hangs at that stage? I can confirm that the Calibre GUI is not running when I execute the backup command inside the container.

Any insights would be very helpful! Thank you!

Edit: confirming that the Calibre Library directory is on an ext4 partition. It consistently fails at the metadata.db file. Wonder if there's a way to run a repair?

Since I'm not going to read back in this thread, are you running calibre from the official calibre site or are you running calibre from a repository?

[shayaknyc]

Originally Posted by JSWolf

Since I'm not going to read back in this thread, are you running calibre from the official calibre site or are you running calibre from a repository?

Originally, I was just running this locally. I had a backup script I would run every 12 weeks or so, that would automatically create a date-named folder, perform the export, and run a hash against all the files. One day, this just stopped working. This thread was my attempt at trying to figure out what could be causing this issue.

I've tried sandboxing the environment, running a debug instance, all of them failed at the same place during the export. I decided, let me try a "clean" environment and run calibre command-line entirely within the container, and just map the external volume into the container. It failed at the same spot - backing up the metadata.db file.

Originally Posted by kovidgoyal

No size is irrelevant. As for repairing, you can always dump it to sqlite and rebuild from that, or use calibredb restore_database to rebuild it completely from OPF files, but read the help of that command to see its limitations.

Before I run this command, I'll backup the metadata.db file. I also clicked on Calibre Library>Library Maintenance>Check Library and the results came back that everything is good. So, I'm not sure that doing a restore_database will solve this issue.