Database problem

[zin]

Of course this happened to me before it got backed up. In fact, I was doing some cleanup, fixing some tags, adding some new books, before I moved the whole database to where my auto backup would pick it up. I had originally loaded the software on a crappy laptop just to give it try, and before I knew it, it had 1000 books and everyone in the house was accessing it via wifi. I thought the new network server would be better than a 6 year old laptop where the battery wouldn't hold a charge. I just didn't move quick enough!

I tried setting "Read metadata from file contents" and it made a bigger mess.

So, am I correct in assuming:
1. All secondary author info is only stored in metadata.db
2. The full (not truncated) book name is only stored in metadata.db
3. All book covers are stored as "cover.jpg" and whether the book has a "cover.jpg" is stored in metadata.db

It turns out I never deleted the source book files (thank you, overly large disk drives), so if I add all of them again from there into a clean library, I'll at least get the books and authors back. The pain will be the covers. Any ideas?

Thanks in advance.

[Starson17]

Originally Posted by zin

So, am I correct in assuming:
1. All secondary author info is only stored in metadata.db
2. The full (not truncated) book name is only stored in metadata.db
3. All book covers are stored as "cover.jpg" and whether the book has a "cover.jpg" is stored in metadata.db

Unfortunately, the answer is "Yes." To put it another way, all metadata is stored in the database - metadata.db. If there's metadata stored anywhere else, it's a byproduct of the fact that Calibre accesses files by labeling them with bits of metadata, or because the user has intentionally stored that metadata elsewhere in a backup, opf file or inside a format/file by making a same format conversion.

(More accurately, on point 3, I believe the database does not store whether the book has a "cover.jpg." Calibre just looks to see if it exists.)

[Perkin]

I've just nearly had this problem, but luckily I had a very recent backup.

My solution would be for calibre to also store an *opf file in the directory with each of the entries, storing all the metadata, so it would be able to re-import the data if needed.

See this thread, for details
https://www.mobileread.com/forums/showthread.php?t=99193

[DoctorOhh]

Originally Posted by chaley

Yes, you can recover this way.

If you are careful, before running db integrity you can manually change the authors back to what they were, and the floating-in-space folders will be found and used. This must be done before calibre notices that the folders are missing and changes the db.

This is what I'm getting, correct me if I'm wrong.

For all the talk about making things much much worse, you don't go to a backup unless things are already bad.

Bottom line your saying if you have to go to a backup, restore a complete backup. That most likely an incremental step like restoring just the metadata.db file will prolong your frustration and in the long run cause more head banging or hair pulling.

[zin]

Originally Posted by Starson17

Unfortunately, the answer is "Yes." To put it another way, all metadata is stored in the database - metadata.db. If there's metadata stored anywhere else, it's a byproduct of the fact that Calibre accesses files by labeling them with bits of metadata, or because the user has intentionally stored that metadata elsewhere in a backup, opf file or inside a format/file by making a same format conversion.

(More accurately, on point 3, I believe the database does not store whether the book has a "cover.jpg." Calibre just looks to see if it exists.)

Tonight I will test your comment on point 3. If you're right, you might have just saved me a lot of effort playing with book covers. I essentially have two near-identical file structures: The old one with no metadata.db, but all the cover.jpg files, and the new one with a correct metadata.db and no cover.jpg files. I can copy the old one over the new one (just the jpgs!) and they should go in the correct places. There are more than 1000 cover files, so it's worth a try. I will of course make copies before trying this.

Also, on the backup/restore part of the discussion, just based on my near-catastrophic database problems, I agree with dwanthny completely - the only way to back up this database is to backup the entire database. Disk space is cheap.

Thanks again for the info/advice.

[Dopedangel]

Originally Posted by zin

Tonight I will test your comment on point 3. If you're right, you might have just saved me a lot of effort playing with book covers. I essentially have two near-identical file structures: The old one with no metadata.db, but all the cover.jpg files, and the new one with a correct metadata.db and no cover.jpg files. I can copy the old one over the new one (just the jpgs!) and they should go in the correct places. There are more than 1000 cover files, so it's worth a try. I will of course make copies before trying this.

Also, on the backup/restore part of the discussion, just based on my near-catastrophic database problems, I agree with dwanthny completely - the only way to back up this database is to backup the entire database. Disk space is cheap.

Thanks again for the info/advice.

I personally think calibre should have a built in backup that backs up metadata.db every time you open calibre for lets say five times it won't be foolproof but in case of metadata.db corruption it could ask if you want to use an older version and save many headaches.
I personally use dropbox so thats backs up the metadata.db for me automatically.

[Manichean]

Originally Posted by Dopedangel

I personally think calibre should have a built in backup that backs up metadata.db every time you open calibre for lets say five times it won't be foolproof but in case of metadata.db corruption it could ask if you want to use an older version and save many headaches.
I personally use dropbox so thats backs up the metadata.db for me automatically.

Uhm, it has been said in this thread that replacing your metadata.db with an older version leads to much headache and hairpulling and thus isn't recommended. So I don't see why the developers would include that as an "official" backup method.

[DoctorOhh]

Originally Posted by chaley

No, calibre won't create blank folders.

Ok, folks are starting to not understand things.

Calibre will, in fact, create empty folders, based on the metadata.db file. Try it yourself. Create an empty library, exit calibre, place a large metadata.db file in the completely empty library (mine had 4019 book) in place of the recently created metadata.db and when you start calibre it will populate that empty library with a complete folder structure that matches what the metadata.db file says should be there. There won't be any files but the entire folder structure will be created (see attached).

Originally Posted by chaley

This must be done before calibre notices that the folders are missing and changes the db.

Calibre does not change the metadata.db file to match the existing folders, it changes the existing folders to match the metadata.db file. It will create folders to match the db but not delete or change any existing folders.

Originally Posted by Manichean

Uhm, it has been said in this thread that replacing your metadata.db with an older version leads to much headache and hairpulling and thus isn't recommended. So I don't see why the developers would include that as an "official" backup method.

While I don't think this should be a feature and that doing a full restore from back up is the least problematic method to restore a library, it may not be the quickest, and doing a full restore is not your only option.

If I had been updating my library for a week (since the last full backup) and I get hit by a power glitch that corrupts my metadata.db file. Simply replacing the corrupt metadata.db file with a recent backup from the last time the metadata.db file was updated prior to the power glitch is a quick way to get you back up and running with minimal risk to pulling out any hair. My metadata.db file using Dropbox version 0.8.?? is backed up every time the file is changed.

While there may be items lost, running a database integrity check should point out where any discrepancies lie. There is virtually zero chance of things getting much worse then not working to begin with, which is the state the library would have been in when you tried to restore it. And there is virtually every chance that the restoration will be a smooth one with minimal bumps in the road.

[chaley]

Originally Posted by dwanthny

Calibre will, in fact, create empty folders, based on the metadata.db file. Try it yourself.

Just did. You are right. And I think that calibre should not do this.

While I don't think this should be a feature and that doing a full restore from back up is the least problematic method to restore a library, it may not be the quickest, and doing a full restore is not your only option.

You pays your dues, you takes your choice.

I don't want to deal with someone asking why his/her foot got shot, so I won't provide a loaded gun. Others may make a different decision.

[DoctorOhh]

Originally Posted by chaley

I don't want to deal with someone asking why his/her foot got shot, so I won't provide a loaded gun. Others may make a different decision.

Seems like a reasonable position to me.