corrupted dbase

[hnoto]

I got the database is corrupted message in Calibre today. I followed each of the directions...unplugged, replugged did not work. Deleted media.xml, let it regenerate and reconnected. Also went in and checked db integrity. None of these are working.

Suggestions? The books look fine on my Sony and I REALLY don't want to lose all the metadata changes I've been making on these.

[speakingtohe]

Probably slow and cludgy but you could try saving all of your books to a single folder using save to disk and then change your database location under preferences and add them to the fresh database.

If they are just the books on the Sony then maybe just chnging the database location could fix it. Not too sure about that though.
Stupid question, but you have rebooted your computer?

[edbro]

I get this all the time. For me, simply resetting the device works. Following the instructions from Calibre do not work (for me).

The database that Calibre is referring to is on the device, not the PC.

[ccayer]

Would it be a good idea to have some sort of backup feature? For example, when Calibre closes, it could make a backup of the db, keeping at most x backups?

[edbro]

Originally Posted by ccayer

Would it be a good idea to have some sort of backup feature? For example, when Calibre closes, it could make a backup of the db, keeping at most x backups?

Again, it is not the Calibre database that is corrupt. The problem is the Sony database (media.xml) on the device. Or, at least Calibre has a problem reading the database. I say that because there are no other ill symptoms using the device.

[hnoto]

Last night I reset my device as well as deleted the media.xml from the Sony. (and rebooted my computer.)

Edbro is right, it isn't the Calibre dbase that is the problem. For some reason, Calibre stops reading the Sony dbase file. Today, after some sleep and caffeine, this finally sank in. The metadata changes are in the Calibre files--which are all ok. (I did do a copy of all the folders over to my external HD last night, just in case.)

I think I've fixed it. I shut down Calibre. Used Explorer to delete all the file folders under database>media>books, and also media.xml. Unplugged the Sony from my computer, reset it and turned back on. Then connected it back up, turned on Calibre and it seems to be working fine now. I now have to move the books back over from the Library to the reader, but at least all the metadata changes I've made via Calibre are still there.

[edbro]

Originally Posted by hnoto

The metadata changes are in the Calibre files--which are all ok. (I did do a copy of all the folders over to my external HD last night, just in case.)

Not sure which files you are referring to here. If you are talking about the files in Calibre on your HD, then just copying them is not sufficient unless you also copy over the big metadata.db file that contains all the metadata. The better way to do it is to select all the files in Calibre then do a Save to Disk. This will also copy metadata into each individual folder. You can reimport those and all the tags will be there.

I don't think you had to delete all the files off the Sony. A simple reset works each time for me. After that, all my files and metadata are still there and it will work again with Calibre. If you do want to delete all the files again, there is a format disk option on the device itself that would be better. That will wipe the memory on the device.

[hnoto]

I tried a simple reset a couple of times and it did not work. Thanks for the info about the save to disk...I had a feeling what I did was too simple to be correct.

[DoctorOhh]

Originally Posted by hnoto

I had a feeling what I did was too simple to be correct.

I'm a big fan of using the save to disk feature to back up your books, I even wrote procedures here to do just that. The truth is that the simple way to back up your library according to the creator of Calibre is to copy your library folder to another location. By default the metadata.db file is in the root of the library folder.

[Starson17]

Originally Posted by dwanthny

I'm a big fan of using the save to disk feature to back up your books

I'm not a big fan of that method. The main reason is probably that I don't have a device that Calibre considers to be an ereader, so I can't use Send to Device. I have to use Save to Disk.

Using Save to Disk as a backup method has 4 big drawbacks:

1) You have to leave the option "Save Cover Separately" turned on. This option is hidden in Preferences on the 2nd tab of the fifth option down. Otherwise your covers don't get backed up.

2) You have to leave the option "Save Metadata in OPF File" turned on. This option is also hidden in Preferences on the 2nd tab of the fifth option down. Otherwise your metadata doesn't get backed up.

3) Your recipes aren't backed up.

4) It's hard to automate a backup.

Problems 1 and 2 can be solved by never turning off those options. However, when you have to constantly use Save to Disk to send to your ereader, as I do, those options are a pain. I could live with the fact that they take up space on my SD card, but the OPF file also produces a file that's the same as the ebook except for extension. My browser on the ereader doesn't show the whole filename, making it hard to pick the ebook instead of the opf file.

Problem 3 can be solved by backing up recipes separately, but that's several steps and harder. Problem 4 isn't really solvable AFAICT.

It really is much simpler to just copy the entire Library directory. Plus you can do incremental backups, if you wish, using a synchronization program. It will find your new ebook folders, copy those, and detect the fact that the metadata.db file has changed and copy it. It's all automated for me, and I can leave the two "Save" options off and not worry about recipes.

Of course, whatever works is fine, but you just have to be more careful.

[edbro]

I back up both ways. I do a daily incremental like you suggest. My big fear however is that if the database ever gets corrupted, all my data is in that one big, corrupt metadata.db file. I don't like having all my hard work residing in one file. Because my backup is automated daily, the corrupted db file will be backed up too.

I don't have/use recipes so that's not a concern for me. I have over 2500 entries in my database with all the metadata set the way I like it. Too much hard work to trust to a single database file.

[kovidgoyal]

The next release of calibre contains code that will try pretty hard to recover from a corrupted database.

Nonetheless, if you feel unsafe, you should use save to disk, as Starson suggested.

And a proper backup scheme is one which maintains differentials over at least the past few days/weeks/months

Like my backup scheme maintains the last 4 days, the last 4 weeks and the last 3 months. But of course, that sort of thing is probably too hard to set up.

[Starson17]

Originally Posted by edbro

My big fear however is that if the database ever gets corrupted, all my data is in that one big, corrupt metadata.db file. I don't like having all my hard work residing in one file. Because my backup is automated daily, the corrupted db file will be backed up too.

I solve that issue in two ways. First, my entire disk backup goes back many days. I can return to the last good copy of metadata.db up to 2 months back. Second, before my backup triggers, my backup program runs a batch file. In that batch file I copy metadata.db to metadata.db.1, etc.. I keep 6 copies for the last 6 backups, all the way to metadata.db.6 It's a very simple rename, of metadata.db.1->metadata.db.2, etc. before copying the current metadata.db to metadata.db.1.

I have yet to see a corrupted metadata.db, but if I ever do, I've got all 6 copies backed up every day for months.

I didn't mention it, but I also worry about the changes introduced in the beta version. There are lots of new features, user defined fields, saved searches, etc. and much of that stuff is also stored in metadata.db. That file is pretty small and really should get backed up, regardless of whether you also save opf and jpg files separately.

[DoctorOhh]

Originally Posted by Starson17

It really is much simpler to just copy the entire Library directory.

It looks like we agree.

[Starson17]

Originally Posted by dwanthny

It looks like we agree.

Yes. I posted some disadvantages of using Save to Disk as a backup strategy, but there are advantages that I didn't mention (although others did). The principal one is the distributed nature of the metadata storage. Each book stores its own little bit of Calibre-compatible metadata with the book formats. That makes it easy to bring the book back in to Calibre in someone else's Calibre library, or recreate your library from a subset of books.