Help! Corrupted database problem

Marok · 06-16-2013, 05:21 PM

Hi, I have a large Calibre library (around 21,000 books) which has just refused to open.
I get a 'Calibre library appears to be corrupted' message with an offer to rebuild. This also happened a couple of months back and choosing the rebuild option didn't work. Can someone please tell me of a fix for this?
Many thanks in advance.

The full error message is:-

Traceback (most recent call last):
File "site-packages\calibre\gui2\main.py", line 240, in initialize_db
File "site-packages\calibre\library\database2.py", line 197, in __init__
File "site-packages\calibre\library\database2.py", line 522, in initialize_dynamic
File "site-packages\calibre\library\caches.py", line 1008, in refresh
File "site-packages\calibre\library\sqlite.py", line 315, in run
DatabaseException: unknown string format
RemoteTraceback (most recent call last):
File "site-packages\calibre\library\sqlite.py", line 272, in run
File "site-packages\calibre\library\sqlite.py", line 172, in get
File "site-packages\calibre\library\sqlite.py", line 40, in _c_convert_timestamp
File "site-packages\calibre\utils\date.py", line 80, in parse_date
File "site-packages\python_dateutil-1.5-py2.7.egg\dateutil\parser.py", line 697, in parse
File "site-packages\python_dateutil-1.5-py2.7.egg\dateutil\parser.py", line 303, in parse
ValueError: unknown string format

Adoby · 06-16-2013, 06:46 PM

Assuming that the metadata backup has had time to save all metadata to opf-files in each folder:

Just create a new library and add all the books in the old folder to it. One book per folder, and read metadata from inside the books, not from the filenames. I think that should work and you would get all the old metadata and formats back?

derangedhermit · 06-17-2013, 12:06 AM

I "write books to disk" for the whole library every so often (especially after a number of books have been added) and then copy that new directory tree to a NAS sitting on our home LAN. I think of it as doing a backup, and am old enough to know that backups can be worth the trouble.

Marok · 06-17-2013, 03:34 AM

Adoby, thank you for the quick response.
This sounds like I might be able to get my baby back with all the lovingly-entered series info and book ratings?
A couple of things I'm unclear about though:-

1. How do I get Calibre to start (neither of the 32 or 64 bit versions that I have will do so)?
2. When you say "Just create a new library and add all the books in the old folder to it. One book per folder, and read metadata from inside the books, not from the filenames.", how do I tell Calibre to read from inside the books?

Thanks again.

Adoby · 06-17-2013, 04:18 AM

Move the current library to some other place, or rename the calibre library folder. Then calibre should start and create a new empty library. (Haven't tried it myself...)

Goto preferences in the GUI. There you can specify if metadata should be read from the filename or not, while adding books. A tick box.

And when you add books from the old calibre library folder, use method 2, Add books from directories, including sub-directories (One book per directory, assumes every ebook file is the same book in a different format).

I don't think that this will restore custom columns, but hopefully "standard" metadata will be restored.

Marok · 06-17-2013, 05:04 AM

Adoby, many thanks for the help.
I've set it going and at the current rate it looks like it will take about 5 hours to complete the rebuild.
I will report back at the (hopefully successful) conclusion.
Thanks again.

Marok · 06-17-2013, 12:39 PM

Phew!
It's back, complete with all series data etc.
Adoby, you're a star. Thank you.

Adoby · 06-17-2013, 02:48 PM

Very nice!

DoctorOhh · 06-18-2013, 12:14 AM

Quote:

Originally Posted by Marok

Phew!
It's back, complete with all series data etc.

Now it is time to back up your library so you don't have to go through this again.

Marok · 06-18-2013, 08:55 AM

Quote:

Originally Posted by DoctorOhh

Now it is time to back up your library so you don't have to go through this again.

DoctorOhh, I did have a backup, from about a month ago, but it takes so long to bring the backup up to date again, it's quicker to rebuild the corrupted database (obviously as long as it's possible to do so).
The last time this happened I went through the 'updating the backup' route with books I'd added since the backup(about 3 weeks), and it took me 2 frustrating weeks.

Adoby · 06-18-2013, 11:38 AM

It seems that there also is a command line utility that can be used to restore a corrupt database using the saved metadata in Opf-files. That way you don't have to create a new library and add the books, might be faster but the end result should be identical.

http://manual.calibre-ebook.com/cli/...store-database

lostinlinux · 06-24-2013, 07:37 AM

guys,
dont know if im pushing my luck here but i have moved all my books on to my synology nas (4000 books) and mapped the server share via nfs to the original library location.
I have also mapped the same share to my wifes windows machine which reads the books and the database fine but the calibre on my linux machine does not seem to accept the library and calibre freezes. i have checked the obvious and the folder is shared with permissions as it was before. Can anyone tell me why calibre wont start on my linux machine or is this one of those times where life in windows is easier because its so lax

Adoby · 06-24-2013, 09:01 AM

Using the same calibre library alternating under Windows and Linux will lead to trouble. Most likely that is what you have discovered. Even using calibre on a networked filesystem may lead to trouble, and is actively discouraged by the developer. Don't risk it without a good backup system and heavy testing.

Cut your losses and try to salvage what you can from the library as soon as possible.

That said, I have my calibre library on a Synology NAS, and mount using NFS from my Ubuntu desktop, and it works fine. But I don't use it from any other computer or operating system.

Check under related tools. I belive some here use the webbserver on their Synology NAS to share their calibre library using different tools, like COPS or Calibre2opds.

06-16-2013, 05:21 PM	#1
Marok Member Posts: 21 Karma: 12376 Join Date: Oct 2010 Device: kindle G3	Help! Corrupted database problem Hi, I have a large Calibre library (around 21,000 books) which has just refused to open. I get a 'Calibre library appears to be corrupted' message with an offer to rebuild. This also happened a couple of months back and choosing the rebuild option didn't work. Can someone please tell me of a fix for this? Many thanks in advance. The full error message is:- Traceback (most recent call last): File "site-packages\calibre\gui2\main.py", line 240, in initialize_db File "site-packages\calibre\library\database2.py", line 197, in __init__ File "site-packages\calibre\library\database2.py", line 522, in initialize_dynamic File "site-packages\calibre\library\caches.py", line 1008, in refresh File "site-packages\calibre\library\sqlite.py", line 315, in run DatabaseException: unknown string format RemoteTraceback (most recent call last): File "site-packages\calibre\library\sqlite.py", line 272, in run File "site-packages\calibre\library\sqlite.py", line 172, in get File "site-packages\calibre\library\sqlite.py", line 40, in _c_convert_timestamp File "site-packages\calibre\utils\date.py", line 80, in parse_date File "site-packages\python_dateutil-1.5-py2.7.egg\dateutil\parser.py", line 697, in parse File "site-packages\python_dateutil-1.5-py2.7.egg\dateutil\parser.py", line 303, in parse ValueError: unknown string format

06-17-2013, 05:04 AM	#6
Marok Member Posts: 21 Karma: 12376 Join Date: Oct 2010 Device: kindle G3	Adoby, many thanks for the help. I've set it going and at the current rate it looks like it will take about 5 hours to complete the rebuild. I will report back at the (hopefully successful) conclusion. Thanks again. Last edited by Marok; 06-17-2013 at 07:47 AM.

06-24-2013, 07:37 AM	#12
lostinlinux Junior Member Posts: 3 Karma: 10 Join Date: Jun 2013 Device: kobo mini	Library help - calibre not starting after change in library guys, dont know if im pushing my luck here but i have moved all my books on to my synology nas (4000 books) and mapped the server share via nfs to the original library location. I have also mapped the same share to my wifes windows machine which reads the books and the database fine but the calibre on my linux machine does not seem to accept the library and calibre freezes. i have checked the obvious and the folder is shared with permissions as it was before. Can anyone tell me why calibre wont start on my linux machine or is this one of those times where life in windows is easier because its so lax

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Corrupted database	Qylie	Calibre	8	08-28-2014 02:02 AM
How to manually fix a (probably) corrupted database?	Lucas Malor	Kobo Reader	13	05-09-2013 08:01 PM
[Old Thread] Corrupted library database	smarties86	Calibre	2	09-02-2012 06:30 AM
Corrupted database	garyft	Calibre	3	05-13-2011 02:10 PM
Grrr... Corrupted Database	PeterT	Kobo Reader	4	01-09-2011 10:36 PM

06-16-2013, 06:46 PM	#2
Adoby Handy Elephant Posts: 1,736 Karma: 26785668 Join Date: Dec 2009 Location: Southern Sweden, far out in the quiet woods Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus	Assuming that the metadata backup has had time to save all metadata to opf-files in each folder: Just create a new library and add all the books in the old folder to it. One book per folder, and read metadata from inside the books, not from the filenames. I think that should work and you would get all the old metadata and formats back?

06-17-2013, 12:06 AM	#3
derangedhermit Addict Posts: 239 Karma: 1280000 Join Date: Oct 2010 Location: USA Device: None	I "write books to disk" for the whole library every so often (especially after a number of books have been added) and then copy that new directory tree to a NAS sitting on our home LAN. I think of it as doing a backup, and am old enough to know that backups can be worth the trouble.

06-17-2013, 03:34 AM	#4
Marok Member Posts: 21 Karma: 12376 Join Date: Oct 2010 Device: kindle G3	Adoby, thank you for the quick response. This sounds like I might be able to get my baby back with all the lovingly-entered series info and book ratings? A couple of things I'm unclear about though:- 1. How do I get Calibre to start (neither of the 32 or 64 bit versions that I have will do so)? 2. When you say "Just create a new library and add all the books in the old folder to it. One book per folder, and read metadata from inside the books, not from the filenames.", how do I tell Calibre to read from inside the books? Thanks again.

06-17-2013, 04:18 AM	#5
Adoby Handy Elephant Posts: 1,736 Karma: 26785668 Join Date: Dec 2009 Location: Southern Sweden, far out in the quiet woods Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus	Move the current library to some other place, or rename the calibre library folder. Then calibre should start and create a new empty library. (Haven't tried it myself...) Goto preferences in the GUI. There you can specify if metadata should be read from the filename or not, while adding books. A tick box. And when you add books from the old calibre library folder, use method 2, Add books from directories, including sub-directories (One book per directory, assumes every ebook file is the same book in a different format). I don't think that this will restore custom columns, but hopefully "standard" metadata will be restored.

06-17-2013, 12:39 PM	#7
Marok Member Posts: 21 Karma: 12376 Join Date: Oct 2010 Device: kindle G3	Phew! It's back, complete with all series data etc. Adoby, you're a star. Thank you.

06-17-2013, 02:48 PM	#8
Adoby Handy Elephant Posts: 1,736 Karma: 26785668 Join Date: Dec 2009 Location: Southern Sweden, far out in the quiet woods Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus	Very nice!

06-18-2013, 11:38 AM	#11
Adoby Handy Elephant Posts: 1,736 Karma: 26785668 Join Date: Dec 2009 Location: Southern Sweden, far out in the quiet woods Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus	It seems that there also is a command line utility that can be used to restore a corrupt database using the saved metadata in Opf-files. That way you don't have to create a new library and add the books, might be faster but the end result should be identical. http://manual.calibre-ebook.com/cli/...store-database

06-24-2013, 09:01 AM	#13
Adoby Handy Elephant Posts: 1,736 Karma: 26785668 Join Date: Dec 2009 Location: Southern Sweden, far out in the quiet woods Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus	Using the same calibre library alternating under Windows and Linux will lead to trouble. Most likely that is what you have discovered. Even using calibre on a networked filesystem may lead to trouble, and is actively discouraged by the developer. Don't risk it without a good backup system and heavy testing. Cut your losses and try to salvage what you can from the library as soon as possible. That said, I have my calibre library on a Synology NAS, and mount using NFS from my Ubuntu desktop, and it works fine. But I don't use it from any other computer or operating system. Check under related tools. I belive some here use the webbserver on their Synology NAS to share their calibre library using different tools, like COPS or Calibre2opds.

Advert

Advert