Corrupted Database

[chilady1]

Couple of days ago, Calibre shut down without warning and when I re-opened it, I got a corrupted database message. I was able to successfully restore the database but I have noticed something weird.

When I click on a specific title within Calibre, the are no formats showing. I typically keep both mobi and epub for all of my books. When I click on the path and it opens the Calibre folder for that specific book, I see both formats along with a cover.jpg and metadata.opf file which is what I would expect.

Why if the files exist in the folder are they not showing in Calibre? I hope this question makes sense and more importantly, how do I get the formats to show back up in the database? I hope someone tells me that I don't have do each and every file individually because if I do - well that is definitely going to take some time.

One final question: is there a limit to the number of books one can have in Calibre? Is there some threshold or ceiling at which this database starts to go crazy? Just wanted to check.

[theducks]

Originally Posted by chilady1

Couple of days ago, Calibre shut down without warning and when I re-opened it, I got a corrupted database message. I was able to successfully restore the database but I have noticed something weird.

When I click on a specific title within Calibre, the are no formats showing. I typically keep both mobi and epub for all of my books. When I click on the path and it opens the Calibre folder for that specific book, I see both formats along with a cover.jpg and metadata.opf file which is what I would expect.

Why if the files exist in the folder are they not showing in Calibre? I hope this question makes sense and more importantly, how do I get the formats to show back up in the database? I hope someone tells me that I don't have do each and every file individually because if I do - well that is definitely going to take some time.

One final question: is there a limit to the number of books one can have in Calibre? Is there some threshold or ceiling at which this database starts to go crazy? Just wanted to check.

The Library 'formats' (the label you see in details) are stored in the (wait for it)...
Database which is why things go wonkers if you delete formats using the OS. The DB still has them listed.


If you select a title: Can you Tap 'O' and does the folder show the book (all formats,cover and OPF)?

[Dopedangel]

Originally Posted by theducks

The Library 'formats' (the label you see in details) are stored in the (wait for it)...
Database which is why things go wonkers if you delete formats using the OS. The DB still has them listed.


If you select a title: Can you Tap 'O' and does the folder show the book (all formats,cover and OPF)?

Did you even read the OPs post that is what she is doing. Where is it mentioned that she accessed the files directly instead of through calibre. She only went to the folder to see if the files are really missing.

@chilady1 try the library maintenance check library and see if it throws up any errors. You should be able to fix most of them.

[chilady1]

Originally Posted by Dopedangel

Did you even read the OPs post that is what she is doing. Where is it mentioned that she accessed the files directly instead of through calibre. She only went to the folder to see if the files are really missing.

@chilady1 try the library maintenance check library and see if it throws up any errors. You should be able to fix most of them.

Thanks! I will try this and see what errors are given. Is there any particular error I should focus on that might restore the files so the format shows back in Calibre?

[Adoby]

There are three main ways to get problems with the calibre library.

1. Operator error, that is to access and do changes to the library directly instead of via the GUI. This includes overwriting the current library or database with a previous backup, instead of fully replacing the library from the backup. Even very minor changes to a filename means that calibre loses track of the book format.

2. Operating and filsystem differences. For instance Linux cares about the case of characters in filenames and paths, Windows doesn't. If you move a calibre library between two operating systems bad things can happen, typically exactly like you described it. Differences in case of the letters in the filename is another example of minor differences that makes calibre lose track of book formats.

3. Hardware and filsystem corruption. All harddrives fail if you use them long enough. Also it is not safe to store the calibre library on a networked drive.

You didn't provide enough information to decide which one is most likely, but perhaps 3 followed by 1?

Some of these problems can be fixed using the filesystem tools in your operating system, in combination with the calibre library maintenance tools. If that doesn't work, then you may have to restore the most recent backup of your valuable library and manually add all the books you added since then. Also redo any other changes.

http://manual.calibre-ebook.com/gui.html#library

All errors that are indicated by the calibre library maintenance tools should be fixed.

[chilady1]

I really appreciate all the good advice offered here but again, is there anyway to get the books so the formats show in Calibre without having to do this one at a time?

[Adoby]

Yes. It could be possible if the library maintenance tools successfully discovers the "missing" formats and offers to add them to the library. However, you may end up with duplicate book records. It depends on what the original cause was and how serious it is. You may also end up with books, formsts or metadata missing from the library.

Could be prudent to restore a backup and compare the number of books to see if any are missing.

[chilady1]

Originally Posted by Adoby

Yes. It could be possible if the library maintenance tools successfully discovers the "missing" formats and offers to add them to the library. However, you may end up with duplicate book records. It depends on what the original cause was and how serious it is. You may also end up with books, formsts or metadata missing from the library.

Could be prudent to restore a backup and compare the number of books to see if any are missing.

Adoby - in Dropbox I see a file called metadata_pre_restore.db - I am assuming this is the database pre-restore. How can I use this file restore my database and as you suggested do a compare?

Thanks!

[BetterRed]

@chilady1 - I think that metadata_pre_restore.db file was created when you ran the Library Maintenance->Restore database function as per your original post.

Originally Posted by chilady1

... Calibre shut down without warning and when I re-opened it, I got a corrupted database message. I was able to successfully restore the database but I have noticed something weird...

Have you run the Library Maintenance->Check Library function as suggested above by Dopedangel. It runs in two phases, the first phase checks the database integrity (should be OK since you just rebuilt it via the Restore database), the second optional phase finds mismatches between the database and the library author/book folders - see attachment for sample output. The Help text on the right explains each condition and the choices you have.

Aside : If this happened to me then I would consider 'severing' the Dropbox link and re-establishing it when I had a pristine clean local library. I can't give you any precise advice on that because I don't use Dropbox.

BR

[Adoby]

I suspect that the use of Dropbox is the cause of the corrupt database.

To avoid problems you must allow Dropbox to fully sync before you start calibre and after you exit calibre, before you shut down the computer. And of course you should not run calibre on more than one computer simultaneously, if they use the same shared calibre library.

[chilady1]

So here's what I had to do, not pretty but better than going through my entire Calibre db one author at a time.

I have restored the last known "good" Calibre db from November 5th from Time Machine. My system didn't go crazy until the 6th.

I did look at the books and I noticed that many of my very early books have multiple epub copies, so once the database is restored, I will go in a do what many have suggested which is run the Quality Check plug-in for a number of items. For example, I will have an author twice in Calibre that looks like this:

Smith, Jane
Smith_Jane

I noticed that the Quality Check plug-in finds those folders without commas, etc. I really think this boils down to a lack of db maintenance on my part. I thank EVERYONE for some very good suggestions and will let you know how things turn out.

[theducks]

Originally Posted by chilady1

So here's what I had to do, not pretty but better than going through my entire Calibre db one author at a time.

I have restored the last known "good" Calibre db from November 5th from Time Machine. My system didn't go crazy until the 6th.

I did look at the books and I noticed that many of my very early books have multiple epub copies, so once the database is restored, I will go in a do what many have suggested which is run the Quality Check plug-in for a number of items. For example, I will have an author twice in Calibre that looks like this:

Smith, Jane
Smith_Jane

I noticed that the Quality Check plug-in finds those folders without commas, etc. I really think this boils down to a lack of db maintenance on my part. I thank EVERYONE for some very good suggestions and will let you know how things turn out.

You can't just swap DB's in willy-nilly.
The DB is just 1/2 of the Database. The other half is the FOLDERS (and contents) They are a Siamese (sneaked my cat in there ) pair

Rename an Author and the folder that contains that book changes
Rename the Title and the Folder for THAT changes
Add a format (also by conversion) and there is a DB entry for even that.


This is why we keep harping on 'Don't make ANY changes using the OS'
1 tiny case mismatch and it is a broken entry

[eschwartz]

Try creating a new empty library and importing all your books, using:

Add books ==> Add books from directories, including subdirectories (one book per directory, assumes every ebook file is the same book in a different format)

This will fix any problems with moved records that the multiple versions of the database can't keep track of. It should also only import one epub per record, and list all multiples with an option to either add or ignore. All metadata changes will be read from the metadata.opf not the files.

I use this if I didn't save to disk on linux but want to import into Windows anyway -- a library is basically a save-to-disk backup except that its a REALLY bad idea to touch with your ten-foot-pole -- it's very handy.

Granted, you will have to re-import all your books, but it can do that without your input while you do something else.

Also, if you have custom columns, use "copy structure from existing library" when creating the new library, so the import will save all your custom metadata.

Then, if you like the results, you can delete the old library.

[chilady1]

Thanks - I was able to successfully restore my Calibre library from 11/4 from Time Machine and now I am in the process of doing some very overdue library maintenance. I really appreciate all of the suggestions here. I am breathing again - WHEW!

[chilady1]

Originally Posted by theducks

You can't just swap DB's in willy-nilly.
The DB is just 1/2 of the Database. The other half is the FOLDERS (and contents) They are a Siamese (sneaked my cat in there ) pair

Rename an Author and the folder that contains that book changes
Rename the Title and the Folder for THAT changes
Add a format (also by conversion) and there is a DB entry for even that.


This is why we keep harping on 'Don't make ANY changes using the OS'
1 tiny case mismatch and it is a broken entry

I don't understand anything you have said above - can you explain what you mean? I really do want to get Smith, Linda and Smith_Linda resolved so I don't have two separate folders for the same author. Can someone explain to me how to accomplish this?

NOTE: I have found the Quality Check plug-in takes care of the above, it finds duplicate author folders and consolidates them and uses the new naming convention with a comma not the underscore _.