Serious problem with Calibre metadata.db

[sja45uk]

I have a serious problem with Calibre 0.9.2 on my Win7 laptop.

I have attempted to reconstruct from notes the following as my attempted recovery sequence but cannot guarantee the order of commands in detail.

I was actively using Calibre when the laptop crashed. I suspect something caused a hard disk problem which caused Windows to reboot. I do not know if it was caused by Windows, Calibre or the laptop hardware as the laptop boot complained it could not find the Operating System but Windows correctly started after a power-off and power-on.

I then used the standard Windows Check Disk on reboot to both: Automatically fix file system errors, and; Scan for and attempt recovery of bad sectors.

Then I tried to run Calibre but I got a calibre.exe is not responding error.

I then tried calibre-debug.exe -g

calibre 0.9.2
Windows-7-6.1.7601-SP1
<'Windows', '7', '6.1.7601'>
Python 2.7.3
Windows: <'7', '6.1.7601', 'SP1', 'Multiprocessor Free'>
Starting up...

Catostrophic failure initializing GUI, bailing out...

I also got a popup window containing the following information:

calibre, version 0.9.2
ERROR: Failed: Restoring database failed, click Show details to see details

Traceback (most recent call last):
File "site-packages\calibre\library\restore.py", line 116, in run
File "site-packages\calibre\library\restore.py", line 279, in replace_db
WindowsError: [Error 32] The process cannot access the file because it is being used by another process

I thought that the calibre installation might be corrupt so I used the file calibre-0.9.2.msi to perform a repair.

At some point calibre asked me the following:

Choose a location for your new calibre e-book library

In doubt of the correct answer, I cancelled out and it has never been asked again so I am not sure which piece of software was asking it as I have performed the calibre installation repair more than once.

Then I tried the following command:
calibredb restore_database --really-do-it

Restoring database succeeded
old database saved as C:\Users\Steve\Calibre Library\metadata_pre_restore.db
Some errors occurred.
A detailed report was saved to calibre_db_restore_report.txt

This showed three different categories of minor problem:

One instance of:
File "site-packages\calibre\library\restore.py", line 159, in scan_library
File "site-packages\calibre\library\restore.py", line 181, in process_dir
File "site-packages\calibre\ebooks\metadata\opf2.py", line 564, in __init__
File "xpath.pxi", line 455, in lxml.etree.XPath.__call__ (src/lxml/lxml.etree.c:115107)
File "apihelpers.pxi", line 37, in lxml.etree._documentOrRaise

(src/lxml/lxml.etree.c:11215)
TypeError: Invalid input object: NoneType

Two instances of:
File "site-packages\calibre\library\restore.py", line 245, in restore_books
File "site-packages\calibre\library\restore.py", line 268, in restore_book
File "site-packages\calibre\library\sqlite.py", line 313, in run
IntegrityError: columns book, format are not unique

Eleven instances of:
File "site-packages\calibre\library\restore.py", line 245, in restore_books
File "site-packages\calibre\library\restore.py", line 259, in restore_book
File "site-packages\calibre\library\database2.py", line 3352, in create_book_entry
File "site-packages\calibre\library\sqlite.py", line 313, in run
IntegrityError: PRIMARY KEY must be unique

Not bad for a database that before the crash contained more than ten thousand books and nothing to indicate any other problems. However, calibre still will not start so I tried another database repair command:

C:\Users\Steve\Calibre Library> calibre-debug --reinitialise-db metadata.db

Python function terminated unexpectedly
disk I/O error
RemoteTraceback (most recent call last):
File "site-packages\calibre\library\sqlite.py", line 248, in run
File "site-packages\calibre\library\sqlite.py", line 226, in connect
OperationalError: disk I/O error (Error Code: 1)
Traceback (most recent call last):
File "site.py", line 132, in main
File "site.py", line 109, in run_entry_point
File "site-packages\calibre\library\debug.py", line 260, in main
File "site-packages\calibre\library\debug.py", line 260, in main
File "site-packages\calibre\library\sqlite.py", line 366, in connect
calibre.library.sqlite.DatabaseException: disk I/O error
RemoteTraceback (most recent call last):
File "site-packages\calibre\library\sqlite.py", line 248, in run
File "site-packages\calibre\library\sqlite.py", line 226, in connect
OperationalError: disk I/O error (Error Code: 1)

The metadata.db and metadata_pre_restore.db are both the same size (20,660 KB), but if I examine them with SQLite Database Browser 2.0 b1.exe metadata_pre_restore.db shows content (although it only contains about 4000 books) but metadata.db appears to be empty.

When I try to run calibre it still hangs and error.txt contains the following:
Python function terminated unexpectedly
disk I/O error
RemoteTraceback (most recent call last):
File "site-packages\calibre\library\sqlite.py", line 248, in run
File "site-packages\calibre\library\sqlite.py", line 226, in connect
OperationalError: disk I/O error (Error Code: 1)

However, I can copy metadata.db using Windows Explorer so I do not understand the error or what I can do next.

Is it an sqlite error and what does it mean?

The permissions appear to be identical for both metadata .db files, so I cannot think of any more fixes to try without more knowledge of the errors and expected recovery behaviour. Thanks, and I apologize for the length of this account of a so-far unsuccessful recovery.

[sja45uk]

REBUILD DATABASE RESOLVED
======================

Spoiler:

Calibre Database Restore
------------------------
C:\Users\Steve> "C:\Program Files\Calibre2\calibredb.exe" restore_database --really-do-it
...
All Wrapped Up (anthology) ... 100%
Restoring database succeeded
old database saved as C:\Users\Steve\Calibre Library\metadata_pre_restore.db
Some errors occurred. A detailed report was saved to calibre_restore_report.txt

C:\Users\Steve>

Windows Explorer
----------------
C:\Users\Steve\Calibre Library\
metadata_pre_restore.db 14/10/2012 05:11 Data Base File 20,660 KB
metadata.db 14/10/2012 06:23 Data Base File 18,516 KB

Windows Explorer - C:\Users\Steve\
calibre_db_restore_report.txt 14/10/2012 06:23 Text Document 7 KB

Restore Text Document
---------------------
C:\Users\Steve\calibre_db_restore_report.txt

Failed to restore the books in the following folders:

C:\Users\Steve\Calibre Library\<AUTHOR>\<BOOK> Traceback (most recent call last):
File "site-packages\calibre\library\restore.py", line 159, in scan_library
File "site-packages\calibre\library\restore.py", line 181, in process_dir
File "site-packages\calibre\ebooks\metadata\opf2.py", line 564, in __init__
File "xpath.pxi", line 455, in lxml.etree.XPath.__call__ (src/lxml/lxml.etree.c:115107)
File "apihelpers.pxi", line 37, in lxml.etree._documentOrRaise (src/lxml/lxml.etree.c:11215)
TypeError: Invalid input object: NoneType

C:\Users\Steve\Calibre Library\<AUTHOR>\<BOOK> Traceback (most recent call last):
Traceback (most recent call last):
File "site-packages\calibre\library\restore.py", line 245, in restore_books
File "site-packages\calibre\library\restore.py", line 268, in restore_book
File "site-packages\calibre\library\sqlite.py", line 313, in run
IntegrityError: columns book, format are not unique

C:\Users\Steve\Calibre Library\<AUTHOR>\<BOOK> Traceback (most recent call last):
Traceback (most recent call last):
File "site-packages\calibre\library\restore.py", line 245, in restore_books
File "site-packages\calibre\library\restore.py", line 259, in restore_book
File "site-packages\calibre\library\database2.py", line 3352, in create_book_entry
File "site-packages\calibre\library\sqlite.py", line 313, in run
IntegrityError: PRIMARY KEY must be unique

Start Calibre
-------------
Corrupted database - popup window
?
The library database at C:\Users\Steve\calibre Library appears to be corrupted.
Do you want calibre to try and rebuild it automatically?
The rebuild may not be completely successful.
If you say No, a new empty calibre library will be created.

Show details
------------
calibre, version 0.9.2
Corrupted database: The library database at C:\Users\Steve\Calibre Library appears to be

corrupted. Do you want calibre to try and rebuild it automatically? The rebuild may not be

completely successful. If you say No, a new empty calibre library will be created.

Traceback (most recent call last):
File "site-packages\calibre\gui2\main.py", line 228, in initialize_db
File "site-packages\calibre\library\database2.py", line 198, in __init__
File "site-packages\calibre\library\database2.py", line 146, in connect
File "site-packages\calibre\library\sqlite.py", line 366, in connect
DatabaseException: disk I/O error
RemoteTraceback (most recent call last):
File "site-packages\calibre\library\sqlite.py", line 248, in run
File "site-packages\calibre\library\sqlite.py", line 226, in connect
OperationalError: disk I/O error

Yes
---
restoring database from backup

Choose a location for your new calibre e-book library - popup window
(click Calibre Library in left panel)
Folder: Calibre Library
(click Select Folder button)

calibre.exe - popup window (1st)
--------------------------------
calibre.exe has stopped working

Windows is collecting more information about the problem.
This might take several minutes...

calibre.exe - popup window (2nd)
--------------------------------
Do you want to send more information about the problem?

Additional details about what went wrong can help Microsoft create a solution.
(press View Details)

Files that help describe the problem:
C:\Users\Steve\AppData\Local\temp\WER2DA6.tmp.WERI nternalMetadata.xml
C:\Users\Steve\AppData\Local\temp\WER71E7.tmp.appc ompat.txt
C:\Users\Steve\AppData\Local\temp\WER7523.tmp.mdmp

Read our privacy statement online:
http://go.microsoft.com/fwlink/?link...8&clcid=0x0409

If the online privacy statement is not available, please read our privacy statement offline:
C:\Windows\system32\en-US\erofflps.txt

(press Cancel button)
Both popup windows disappear.

Windows Explorer
----------------
Windows Explorer - C:\Users\Steve\Calibre Library\
metadata.db 14/10/2012 07:01 Data Base File 20,660 KB

ALTERNATIVE
===========

Try starting Calibre without metadata.db file
---------------------------------------------
(pulldown 'Choose calibre library to work with' menu)
(select 'Library Maintenance > Check library' option)

Checking library integrity

Success
?
Found no errors in your calibre library database.
Do you want calibre to check if the files in your library match the information in the

database?

(click 'Yes' button)
--------------------
Calibre starting library scan - this may take some time
Check Library -- Problems found

Name Path from library
> Invalid titles
Unknown Unknown\Unknown - Unknown.pdf
_Archive _Archive\metadata.db-journal
_Archive _Archive\meta_db_prefs_backup.json
> Extra titles
(very long list in alphabetical order of <AUTHOR>\<BOOK>)
> Invalid authors
error.txt error.txt
metadata - Copy.db metadata - Copy.db
metadata_pre_restore.db metadata_pre_restore.db
> Extra authors
(very long list in alphabetical order of <AUTHOR>)
Missing book formats
Extra book formats
Unknown files in books
Missing covers files
Cover files not in database
Folders raising exceptions

Names to ignore: _Archive

Extensions to ignore

Help

calibre stores the list of your books and their metadata in a database. The actual book files

and covers are stored as normal files in the calibre library folder. The database contains a

list of the files and covers belonging to each book entry. This tool checks that the actual

files in the library folder on your computer match the information in the database.

The result of each type of check is shown to the left. The various checks are:
Invalid titles: These are files and folders appearing in the library where books titles should,

but that do not have the correct form to be a book title.
Extra titles: These are extra files in your Calibre library that appear to be correctly-formed

titles, but have no corresponding entries in the database

* Invalid authors: These are files appearing in the library where only author folders should

be.

* Extra authors: These are folders in the Calibre library that appear to be authors but that

do not have entries in the database

* Missing book formats: These are book formats that are in the database but have no

corresponding format file in the book's folder.

* Extra book formats: These are book format files found in the book's folder but not in the

database.

* Unknown files in books: These are extra files in the folder of each book that do not

correspond to a known format or cover file.

* Missing cover files: These represent books that are marked in the database as having covers

but the actual cover files are missing.

* Cover files not in database: These are books that have cover files but are marked as not

having covers in the database.

* Folder raising exception: These represent folders in the Calibre library that could not be

processed/understood by this tool.

There are two kinds of automatic fixes possible: Delete marked and Fix marked.

Delete marked is used to remove extra files/folders/covers that have no entries in the

database. Check the box next to the item you want to delete. Use with caution.

Fix marked is applicable only to covers and missing formats (the three lines marked 'fixable').

In the case of missing cover files, checking the fixable box and pushing this button will tell

calibre that there is no cover for all of the books listed. Use this option if you are not

going to restore the covers from a backup. In the case of extra cover files, checking the

fixable box and pushing this button will tell Calibre that the cover files it found are correct

for all the books listed. Use this when you are not going to delete the file(s). In the case of

missing formats, checking the fixable box and pushing this button will tell calibre that the

formats are really gone. Use this if you are not going to restore the formats from a backup.

(press 'Copy to clipboard - button)

(press 'Done' button)

Calibre window
--------------
(pulldown 'Choose calibre library to work with' menu)
(select 'Library Maintenance > Restore database' option)

Are you sure?
?
Your list of books, with all their metadata is stored in a single file, called a database. In

addition, metadata for each individual book is stored in that books' folder, as a backup.

This operation will rebuild the database from the individual book metadata. This is useful if

the database has been corrupted and you get a blank list of books.

Do you want to restore the database?

(click 'Yes' button)

Restoring database - popup window
---------------------------------
Restoring database from backups, do not interrupt, this will happen in three stages...

Windows Explorer
----------------
C:\Users\Steve\Calibre Library\
metadata_pre_restore.db 14/10/2012 07:40 Data Base File 116 KB
metadata.db 14/10/2012 09:02 Data Base File 18,516 KB

SOLUTION
--------
For a successful database rebuild, deleting the metadata.db was essential before doing the database rebuild. The following failed to delete or rename this file, so was destined to be unsuccessful with whatever was the cause of this problem. However, I have not determined whether the two ways of rebuild the database are exactly equivalent (although I assume they are).

calibredb restore_database --really-do-it

In addition, it appears the the term restore_database is misleading because it is really a rebuild_database operation as it does not recover the database from a single archive file.

[Divingduck]

Sorry, but your post is a horror for reading. A better way is to use Spoiler for your log listings. Then it is better to see what is a log and what is something what you wrote as comment or question.

If Calibre is not able to override the metadata.db then your OS is blocking the file. These mainly happen when an application is crashing without unlock the file. In this case you need to shut down the user profile and restart again. Then you can rename or delete the metadata.db and start Calibre again and rebuild the database.

[sja45uk]

Thanks for the Spoiler link but it looks like there is a serious learning curve to improve the log with the tags. What I really need is some form of utility to capture the data as I encounter the bug.

I think that my problems were caused by Calibre running in the System Tray which means that closing Calibre and restarting it was not sufficient. I have learned the lesson to log out and log back in to clear all running applications. However, it is good to know that I do not have to scrabble around for a lock file like I have to do with VMware and other virtualisation software.

[theducks]

Originally Posted by sja45uk

Thanks for the Spoiler link but it looks like there is a serious learning curve to improve the log with the tags. What I really need is some form of utility to capture the data as I encounter the bug.

I think that my problems were caused by Calibre running in the System Tray which means that closing Calibre and restarting it was not sufficient. I have learned the lesson to log out and log back in to clear all running applications. However, it is good to know that I do not have to scrabble around for a lock file like I have to do with VMware and other virtualisation software.

When running with the system Tray ON: You can Quit by right-clicking on the tray Icon and making the selection there

[itimpi]

Originally Posted by theducks

When running with the system Tray ON: You can Quit by right-clicking on the tray Icon and making the selection there

Or you can use CTRL-Qf rom within Calibre which seems to quit Calibre regardless of whether you have System Tray on or off.