recovering a very large Calibre library after system crash

[lucask]

I’m running Calibre 8.4 on Linux and have a very large library I’ve been maintaining for over a decade (around 19k files, ~180 GB). My laptop recently crashed and my only option now is to reinstall the OS.

Using a live USB, I was able to launch Calibre and use the Export Library tool to back up the library in multiple .calibre-data parts, which I’m currently uploading to Dropbox. Before I wipe the drive, I’d like some guidance:

- If in my new Calibre install the import fails for some reason, can I manually recover the old directory structure from all the .calibre-data files?
- Is there a reliable way to verify that the Dropbox uploads are complete and uncorrupted?
- ISince I don’t have access to another machine to test the restore, is there any way to confirm in advance that the exported parts will restore correctly?

More broadly, what would you do in my position?

[PeterT]

Rather than using calibre export, seeing that you have access to the files right now, why not simply copy the entire calibre library directory tree to another USB device, along with the calibre preferences (located in .config/calibre and all its sub directories); then it's simply a matter of reinstalling calibre, restoring it's preferences and library, and then starting calibre.

[JSWolf]

Originally Posted by lucask

I’m running Calibre 8.4 on Linux and have a very large library I’ve been maintaining for over a decade (around 19k files, ~180 GB). My laptop recently crashed and my only option now is to reinstall the OS.

Using a live USB, I was able to launch Calibre and use the Export Library tool to back up the library in multiple .calibre-data parts, which I’m currently uploading to Dropbox. Before I wipe the drive, I’d like some guidance:

- If in my new Calibre install the import fails for some reason, can I manually recover the old directory structure from all the .calibre-data files?
- Is there a reliable way to verify that the Dropbox uploads are complete and uncorrupted?
- ISince I don’t have access to another machine to test the restore, is there any way to confirm in advance that the exported parts will restore correctly?

More broadly, what would you do in my position?

Don't rely on Dropbox as your only backup. And don't rely on an export as your only backup. I've read of some people having problems importing an export. Your best solution would be to get a USB stick or external hard drive of large enough size and backup your entire drive. In most cases, when you install the OS, you don't have to wipe the drive as long as the drive is OK. So you'd have a full backup and maybe no need to restore the backup.

[lucask]

Originally Posted by PeterT

Rather than using calibre export, seeing that you have access to the files right now, why not simply copy the entire calibre library directory tree to another USB device, along with the calibre preferences (located in .config/calibre and all its sub directories); then it's simply a matter of reinstalling calibre, restoring it's preferences and library, and then starting calibre.

I don't have a large enough external hard drive at hand, and I can't afford to take this into the shop. I'm really worried I might only get partial backups randomly scattered if I try copying the whole folder as you suggest. I've got the checksums of all the calibre parts, wouldn't that be less risky? I can then try downloading a couple of parts and seeing whether they match.

[PeterT]

So upload the directories directly to DropBox.

I don't believe there is any way to manually recover files from the calibe export parts.

[kovidgoyal]

Upload the calibre library folder to dropbox as well. The calibre export is of the same size approximately so if you can upload one you can upload the other. Or if for some reason you dont want to do that, use your live usb to restore the exported parts as a test.

[lucask]

Originally Posted by kovidgoyal

Upload the calibre library folder to dropbox as well. The calibre export is of the same size approximately so if you can upload one you can upload the other. Or if for some reason you dont want to do that, use your live usb to restore the exported parts as a test.

Thanks for replying. The advantage I see with the export is that I can make sure all of the parts are there, and I can test random parts with a checksum to be quite certain what I have in my backup is exactly what I have locally. I can't install the dropbox client locally, and can't upload the whole folder in one go so instead I have to drag all the subfolders into the browser in batches. If I skip something or make a mistake I would have no way of telling.

Is it possible to download just a few parts and then attempt an import?

[PeterT]

I could be snarky here and suggest that you just restore your Calibre library and configuration from the backups you make on a regular basis of your important data.

You DO have backups of your important 180 Gb / 19k eBook library, right??

[lucask]

Originally Posted by PeterT

I could be snarky here and suggest that you just restore your Calibre library and configuration from the backups you make on a regular basis of your important data.

You DO have backups of your important 180 Gb / 19k eBook library, right??

I have an old backup in Dropbox, but I still haven't lost my data so I'm hoping I won't have to lose anything.

I'll try Kovid's suggestion to upload and delete the calibre parts locally then trying to redownload everything and import from the live USB. If that works I'll proceed to wipe the hard drive, otherwise I'll start backing up the folders one-by-one.

[kovidgoyal]

No there is no partial import, you need all the parts to be able to import successfully.

[JSWolf]

Originally Posted by lucask

I don't have a large enough external hard drive at hand, and I can't afford to take this into the shop. I'm really worried I might only get partial backups randomly scattered if I try copying the whole folder as you suggest. I've got the checksums of all the calibre parts, wouldn't that be less risky? I can then try downloading a couple of parts and seeing whether they match.

Do you know why your computer crashed and needs the OS reinstalled? If it's the hard drive, then it's on the way out and needs to be replaced.

[Dal Bhat]

Something so valuable should not have only an "old" backup.

[lucask]

I've tested the import and it seems to work. The only difference is that the metadata.opf files in each folder are gone in the new library. As far as I can tell that's perfectly fine, I can just regenerate the metadata backups afterwards.

As for why it crashed, my OS got misconfigured, it's entirely a boot issue. I've been advised the only way is to reinstall everything.

[JSWolf]

Originally Posted by lucask

I've tested the import and it seems to work. The only difference is that the metadata.opf files in each folder are gone in the new library. As far as I can tell that's perfectly fine, I can just regenerate the metadata backups afterwards.

As for why it crashed, my OS got misconfigured, it's entirely a boot issue. I've been advised the only way is to reinstall everything.

Who told you that? There should be no reason why you cannot reinstall the OS without having to totally nuke the drive.

But before you do the install, backup any data you want to keep just in case.

[lucask]

Originally Posted by JSWolf

Who told you that? There should be no reason why you cannot reinstall the OS without having to totally nuke the drive.

But before you do the install, backup any data you want to keep just in case.

Here's where I got that advice: https://discourse.nixos.org/t/nixos-...e-system/71964