invalid database

[aceflor]

Help please !!!!

This morning Calibre for the first time crashed on me. When I tried to restart it, it told me the database was corrupt. Restore did not work, Calibre ends up creating a new empty Library. Here the error message I keep receiving if I try to switch to my Library Folder (where all my books, covers and opf files actually are and have always been):

Traceback (most recent call last):
File "site-packages\calibre\gui2\ui.py", line 636, in library_moved
File "site-packages\calibre\db\legacy.py", line 70, in __init__
File "site-packages\calibre\db\legacy.py", line 44, in create_backend
File "site-packages\calibre\db\backend.py", line 344, in __init__
File "site-packages\calibre\db\backend.py", line 1066, in fget
File "site-packages\calibre\db\backend.py", line 784, in conn
File "site-packages\calibre\db\backend.py", line 243, in __init__
File "site-packages\calibre\db\backend.py", line 287, in execute
File "c:\cygwin64\home\kovid\sw\build\apsw-3.8.2-r1\src\cursor.c", line 990, in APSWCursor_execute.sqlite3_prepare
File "c:\cygwin64\home\kovid\sw\build\apsw-3.8.2-r1\src\statementcache.c", line 386, in sqlite3_prepare
CorruptError: CorruptError: database disk image is malformed

What can I do? I don't really want to reimport manually over 5000 books....

[theducks]

Have you tried renaming metadata.db (just in case) to somethg else, then try a repair?
'Repair' assumes there are metadata.OPF files in each title folder

[eschwartz]

Also, for the future: are you using any syncing mechanisms? This doesn't sound like the usual dropbox issue, I think I have seen that (similar) error once when accidentally accessing the library from two running calibres at the same time, via a linux ltsp setup logging in from two client PCs.

[BetterRed]

and - and do regular library backups - i.e. the library folder in its entirety, not just the database.

If you use an incremental backup regime, then after doing it the first time it should only take a couple of minutes a day.

BR

[aceflor]

I have not tried renaming the metadata.db, that is actually a good idea...

I did not access calibre from 2 places at the same time. My library is on OneDrive, but I am always accessing it from the same one and only laptop.

My last backup is from August, and since then I have cleaned and organized my library. I would love not to loose the work.....

[eschwartz]

Originally Posted by aceflor

I have not tried renaming the metadata.db, that is actually a good idea...

I did not access calibre from 2 places at the same time. My library is on OneDrive, but I am always accessing it from the same one and only laptop.

My last backup is from August, and since then I have cleaned and organized my library. I would love not to loose the work.....

Well, a restore should fix it anyway.

Corruption of the database almost never IME correlates with missing book formats+OPFs.

Dropbox can disappear books via syncing conflicts, leading to duplication of old versions, but normally the database either disappears or remains intact but possibly old. Not corrupted.

tl;dr your files (database aside) are probably all right, so a restore will solve everything.

Prudence would dictate you backup your whole library Just In Case, though.

[theducks]

Originally Posted by aceflor

I have not tried renaming the metadata.db, that is actually a good idea...

I did not access calibre from 2 places at the same time. My library is on OneDrive, but I am always accessing it from the same one and only laptop.

My last backup is from August, and since then I have cleaned and organized my library. I would love not to loose the work.....

http://manual.calibre-ebook.com/faq.html#id43

You hit the case the FAQ was trying to get you to avoid.
'It used to work'

Trans: 'I have been getting away with it (until now)'
NAS/Cloud are fine for backups, but not the Live Library.

[eschwartz]

Originally Posted by theducks

http://manual.calibre-ebook.com/faq.html#id43

You hit the case the FAQ was trying to get you to avoid.
'It used to work'

Trans: 'I have been getting away with it (until now)'
NAS/Cloud are fine for backups, but not the Live Library.

It is risky, but not by any means guaranteed to fail.

Prudence dictates we sync a one-way mirror of the library, but plenty of people do the unwise thing and get away with it, with no reason to think it will stop working... depends on how careful you are with pausing sync while calibre is open. IOW, it too is perfectly fine.

The evidence does not hypothesize on whether that is the problem here. In fact: "My last backup is from August" -- probably not.


This is NOT a NAS situation.

[theducks]

I would expect a NAS (local) to have the better response time.

But the bottom line is both are still Networked drives, subject to things Calibre is not equipped to deal with.

[aceflor]

To be clear : my library is accessible offline. But the last library and metadata.db stand I secured on an external drive is from August. I was actually going to secure it again next week, as my cleaning of the library would have been finished.

[jgaiser]

After *many* disasters I now backup weekly. Week 1 full backup, next 3 weeks incremental. Rinse. Repeat.

[aceflor]

Ok, feeling much better right now. Here for anyone else in my case:

1. Try to restore
2. If restore does not work (it did not for me), try to rename the metadata.db
3. If that does not work (it did not for me), you will have to re-add your books manually
4. To quicken things a bit and not loose the info packed in the various .opf, I zipped books by date, 10 to 20 per zip, and added them using the "add zip" function in Calibre. THAT WORKS.

I am now recovering the library with almost all my books back (some very recent ones seemed problematic and would not zip, which might have been the beginning of the whole story), with their metadata, tags, series, shelves, and for me very important: original date instead of today's date. I am almost done.

I now only need to secure everything on an external drive and find what exactly the problem with those books which would not zip is.

Thanks for every advice, I know I should backup more often, but I live in 3 different countries and keep flying back and forth between those 3 and lately the US, which makes backup difficult and of course the Cloud very attractive.

[eschwartz]

Why did the restore fail?

In any event, if you:
add books from directory including sub-directories, one book per directory, assume each file is the same book in a different format,
you should get metadata from the OPF imported, as well. No need to zip them up.

[aceflor]

I don't know why, but it failed, each time I tried (3 Times in total). I zipped the directories per date, as to be sure to cover every new book since the end of August. My Library shows Directories per Author, and sub-directories per book, and as I suspected some books might be corrupted I had to re-add them methodically to find the bad ones.

3 books turned out to be indeed corrupted.

I recovered all books but those 3, all tags, but some series are missing. Oddly, too, azw3 formats (converted through Calibre) have sometimes disappeared.

I can only remember a transfer between my laptop and my kindle fire done through CC going wrong (as in giving me the corrupt database message), ejecting the fire, exiting Calibre and not being able to start it again correctly.

Now that my library is rescued, I will need to empty the fire and refill it again. Probably not going to use CC this time around, and will not fill the fire before having done a backup of the library anyway.

[chaley]

Originally Posted by aceflor

I can only remember a transfer between my laptop and my kindle fire done through CC going wrong (as in giving me the corrupt database message), ejecting the fire, exiting Calibre and not being able to start it again correctly.

Now that my library is rescued, I will need to empty the fire and refill it again. Probably not going to use CC this time around, and will not fill the fire before having done a backup of the library anyway.

What "corrupt database" message? CC doesn't have a message containing "corrupt" or its variants, so I am guessing that the message came from calibre.

The wireless driver that CC uses does not write the calibre database except under one condition: the "sync read information" column names have been filled in. In this case the driver uses the calibre database layer, just as the rest of calibre does.

Like eschwartz, I am interested in how the restore failure manifested itself. How did you know it failed? Were there error messages? Something else? I wrote that code, so if there is something broken I need to fix it.

Your problems "smell" a bit like they come from failing hardware. Have you checked that the disc in your computer is not going bad or that the file system is messed up? If not, you might want to schedule a full chkdsk (one that reads the entire disc) to see if there are bad spots. The same thing might be happening to the RAM in your machine. A memory tester would put that possibility to bed.