Library crash - unable to rebuild

[Ellachanted]

Hi, I am hoping someone can help.

I have a very large fanfiction library with a number of custom columns. Unfortunately, with no warning, it crashed. I am normally good with updating my back up, but forgot to do it for a few weeks, of course after I did a lot of work cleaning up and marking stories read, etc - stuff I would like to keep.

calibre, version 2.0.0
ERROR: Failed: Restoring database failed, click Show details to see details

Traceback (most recent call last):
  File "site-packages\calibre\db\restore.py", line 125, in run
  File "site-packages\calibre\db\restore.py", line 277, in replace_db
WindowsError: [Error 32] The process cannot access the file because it is being used by another process

I updated Calibre and still got the error. I shut down, then removed Panda Free and went back to Windows Defender. I also turned off the internet access and virus protection before running after the first time I got the errro message. It ran for over 24 hours before giving that message.

I have deleted the metadata.db files, tried creating a new library, moving the metadata.db file to the old library and trying to restore. I got this error message then

Traceback (most recent call last):
  File "site-packages\calibre\gui2\ui.py", line 630, in library_moved
  File "site-packages\calibre\db\legacy.py", line 70, in __init__
  File "site-packages\calibre\db\legacy.py", line 44, in create_backend
  File "site-packages\calibre\db\backend.py", line 344, in __init__
  File "site-packages\calibre\db\backend.py", line 1066, in fget
  File "site-packages\calibre\db\backend.py", line 784, in conn
  File "site-packages\calibre\db\backend.py", line 243, in __init__
  File "site-packages\calibre\db\backend.py", line 287, in execute
  File "c:\cygwin64\home\kovid\sw\build\apsw-3.8.2-r1\src\cursor.c", line 990, in APSWCursor_execute.sqlite3_prepare
  File "c:\cygwin64\home\kovid\sw\build\apsw-3.8.2-r1\src\statementcache.c", line 386, in sqlite3_prepare
CorruptError: CorruptError: database disk image is malformed

So it seems to be dead. I don't know enough out programming to do any fancy stuff.

So my question is, is there any way to get the metadata from the opf files into the back up database? I added all the changed files from the old database to the new database by copying them to another drive (just the changed ones) but the metadata didn't add. I would rather not do it all manually if I don't have to.

And is there a maximum size on the database before it becomes unstable? I have plenty of room on my hard drive.

Thanks for any help!

[kovidgoyal]

First of all update calibre. Then you can run the restore from the command line, that sometimes avoids file access issues caused by antivirus programs.

calibredb restore_database -r

[Ellachanted]

Thanks. As soon as it finishes downloading, I will do that and try to rebuild.

Since I mostly use it to download fanfics and remove DRM from my Kindle books, I usually just update when they require it.

[Ellachanted]

Updated Calibre to the latest version (2.27) 64 bit. Ran the library maintenance restore library and after running about 24 hrs I get this again:

Traceback (most recent call last):
File "site-packages\calibre\db\restore.py", line 125, in run
File "site-packages\calibre\db\restore.py", line 277, in replace_db
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: u'F:\\My documents\\Calibre fanfic\\comp fanfics -a\\metadata.db'


I didn't turn off the virus check this time. I have to shut down the internet to do that. I'm going to try something different first and download the 32 bit version, shut off the virus, and try again and not touch the computer.....

[kovidgoyal]

I suggest booting windows into safe mode, making sure all antivirus/file sync/dropbox programs are not running then running the restore.

[Ellachanted]

I turned off the automatic syncing a while ago when I found it messed with the library. So it no longer runs on startup. My main library also, was not in the syncing folder. (Hey, I learned my lesson after having to rebuild my library after using Google Drive)

I got a "corrupt file" error. So I moved all the files to a new location where I had set up an empty library. I then turned off wifi and virus protection and ran the rebuild. 24+ hours later, I have the majority of my books and data back. Well, the books will always be there, it was the wonderful metadata that I really wanted.

Although it rebuild the majority of the library, there were some errors that appear to all be the same:

Failed to restore the books in the following folders:
F:\My documents\Calibre fanfic\comp fanfics -b\Vixen1836\The Beauty and The Geek (1) with error:
Traceback (most recent call last):
File "site-packages\calibre\db\restore.py", line 253, in restore_books
File "site-packages\calibre\db\cache.py", line 57, in call_func_with_lock
File "site-packages\calibre\db\cache.py", line 1979, in restore_book
File "site-packages\calibre\db\cache.py", line 1487, in create_book_entry
File "site-packages\calibre\db\backend.py", line 804, in execute
File "c:\cygwin64\home\kovid\sw\build\apsw-3.8.2-r1\src\cursor.c", line 231, in resetcursor
ConstraintError: ConstraintError: UNIQUE constraint failed: books.id

So I am guessing somehow they got the same id as another book in the library. It isn't a lot, so I can manually re-add and fix.

I then ran a Library Check and found a lot more extra books and authors. Some may be duplicates that don't matter. Unfortunately, they can't be automatically fixed

So the rest is manual I guess. I copied everything to a notepad file so I have all the info.

I have a backup library that was last updated about a month ago. Is there anyway to compare libraries?

Thanks for all your help. This is such a great program!

[theducks]

You appear to have not read the FAQ and dug yourself into a whole lot of work .

1)Synch OFF (at least the Library folder) while using Calibre
2)Files (books) may not be added to/removed from a Library folder EXCEPT by using Calibre (GUI or CLI). Applies to Calibre Known Items

One way to recover:
Create a New Empty Library (Local Drives only) withe the same structure (Create/Switch has options)
Select All books in existing Library: RIGHT CLICK: Copy to Library:<new> Delete after copy.

What is LEFT OVER in your old Library FOLDER, is what Calibre could/did not know about. Those are now your job to deal with and possibly add (using a normal Calibre Add Books method)

[eschwartz]

Add books ==> Add books from directories, including sub-directories (One book per directory, assumes every ebook file is the same book in a different format)


This will add all metadata from the backup OPF files as well.

[Ellachanted]

Originally Posted by theducks

You appear to have not read the FAQ and dug yourself into a whole lot of work .

1)Synch OFF (at least the Library folder) while using Calibre
2)Files (books) may not be added to/removed from a Library folder EXCEPT by using Calibre (GUI or CLI). Applies to Calibre Known Items

One way to recover:
Create a New Empty Library (Local Drives only) withe the same structure (Create/Switch has options)
Select All books in existing Library: RIGHT CLICK: Copy to Library:<new> Delete after copy.

What is LEFT OVER in your old Library FOLDER, is what Calibre could/did not know about. Those are now your job to deal with and possibly add (using a normal Calibre Add Books method)

So reading your suggestions, this will not help the problem. The original library was corrupted. I got error messages opening it. So copying from it to a new library is not an option. I know how to do that because that is how I send books to my back up library. (highlight/right click/copy to library - no delete because it is the back up)

The copy library was only a back up and never the main library. Additionally, the Copy sync is only on when I turn it on. I do not need it running constantly. The Google fiasco was a couple years ago. And it was more of an attempt to see if Google Drive would work as a back up. But I did learn how to rebuild a library from it.

So the library was on a drive in the computer. This was not the issue. The original library crashed due to a corrupted metadata file. I do not know what caused the corruption. As I said, it was working fine the day before. It is not on an NAS or online drive. A drive check did not show any problems with the drive. However, right before I did the last rebuild, I got a drive check error. So, who knows?

My rebuild attempts following the manual and various posts here were not successful. The new metadata files would give error messages. Calibre couldn't find the new libraries.

I then sorted the main Calibre folder by date and copied the ones dated since my last backup to a temp folder and added that folder to my back up library, which now has a major amount of duplicates. I tried adding directly but got file is corrupt errors. But copying to a new location seemed to work


I then got the bright idea to move the whole library to another empty library folder on the same drive - the library was dead at that point. Multiple efforts to rebuild and restore following the directions had resulted in corruption errors. I tried adding from in place and got error messages. In the new library, I ran restore. It was mostly successful. I have a list of stories I will have to manually add that I obtained from running a library check.

This is a fanfiction library. A lot of the stories are no longer available or I would just rebuild it from the URLs. Though that won't help with tracking what I've read and want to read. Authors also add chapters, delete chapters and add new ones, and generally mess stuff up. So just "deleting" duplicates is not an option. And it is large.


So adding....

Adding only seems to add some of the metadata I've found. Either that or the OPF files are damaged in the ones I have to add. Or I may have to associate the data to a column? I will continue working on it. As I said, at least I haven't lost any books, just my ease of finding things.

But I really appreciate all your help! This has been much easier than using FFDL standalone and an Excel spreadsheet.

[eschwartz]

Adding will not add custom columns unless a matching custom column already exists.

(Make sure you used the option I indicated above.)

[theducks]

It may still help to Fix your rebuilt library by making a new (3rd) and that will leave the remaining problems behind.

Not all solutions can be accomplished in a single step

[Ellachanted]

Originally Posted by eschwartz

Adding will not add custom columns unless a matching custom column already exists.

(Make sure you used the option I indicated above.)

Thanks! I will probably end up adding a number of books this way.

[Ellachanted]

Originally Posted by theducks

It may still help to Fix your rebuilt library by making a new (3rd) and that will leave the remaining problems behind.

Not all solutions can be accomplished in a single step

LOL! I am finding this out.

I was just starting to make a new library, with all the various books from the old libraries and the messed up rebuilt one. Clean it up and start all over.

Well, I really didn't want to clean my house anyway.

[Ellachanted]

I have a question - I am adding books. It states it is Reading Metadata and adding books to library. It says it is 25% done but there are no books listed yet.

Shouldn't they be showing in the screen? They are adding to the folder (checked with windows explorer)

I seem to remember the showed up during this process. But then this may be the most books I've added at once.

[theducks]

Originally Posted by Ellachanted

I have a question - I am adding books. It states it is Reading Metadata and adding books to library. It says it is 25% done but there are no books listed yet.

Shouldn't they be showing in the screen? They are adding to the folder (checked with windows explorer)

I seem to remember the showed up during this process. But then this may be the most books I've added at once.

The GUI is updated at the end of A task.
Add FolderS is 1 task (100% when done)