Calibre Database rebuild

mitch13 · 02-11-2015, 12:12 AM

I have a very large library (in the tens of thousands). All has been going well for a number of years and Calibre performance on this large file is excellent.
While working on my library recently (wanting to transfer all pdf files to a separate library) my PC had a problem and when it had finally recovered from its 'fit' had lost my entire calibre library.
I did have a total backup even though it was a couple of days old and so I copied this library folder to the folder that had held my current library (now non-existent). Calibre then opened with close to the number of books I had.
On running a library check it pronounced the library healthy but when followed by the next stage check I had a large number of books physically in the library but not in the database.
After awhile and getting very nervous I decided to run a Restore Database believing that this would solve the problem. I have tried many times now and always during the second of three stages my PC goes into my least favourite mode ie 'Not Responding'. I then have to kill Calibre via Task Manager.
Debug message:

calibre Debug log
calibre 2.19 [64bit] isfrozen: True is64bit: True
Windows-8-6.2.9200 Windows ('64bit', 'WindowsPE')
('Windows', '8', '6.2.9200')
Python 2.7.9
Windows: ('8', '6.2.9200', '', 'Multiprocessor Free')
Starting up...
Problem creating accessible interface for: CalibreSplashScreen(0x332b3f0)
Make sure to deploy Qt with accessibility plugins.
Problem creating accessible interface for: QScrollBar(0x87f5d80)
Make sure to deploy Qt with accessibility plugins.
Started up in 12.60 seconds with 0 books
QTextCursor::setPosition: Position '1' out of range
QTextCursor::setPosition: Position '382' out of range
QTextCursor::setPosition: Position '87' out of range

Any suggestions to recover my library built over about 4 years!

Regards

mitch13

JSWolf · 02-11-2015, 12:13 AM

No backup?

kovidgoyal · 02-11-2015, 12:18 AM

Just let it run, not responding just means the GUI is frozen not that the actual restore process has failed. This can happen if some restore operation is taking very long, blocking the GUI from responding until it completes.

You can also run the restore from the command line, which removes the possibility of GUI problems.

calibredb restore_database

mitch13 · 02-11-2015, 03:02 AM

Thanks Kovid. One dumb question - where do I run the command line from?

kovidgoyal · 02-11-2015, 03:12 AM

The windows command prompt (cmd.exe)

drjenkins · 02-11-2015, 07:40 AM

Sometimes when a GUI freezes I pull up Task Manager to get some reassurance that the process is still consuming CPU cycles.

mitch13 · 02-12-2015, 12:20 AM

Hi Kovid,
Ran the Command version (DOS) as suggested - took around 20 hours but that not a problem. The command I used was c\: calibredb restore_database "Really-Do-It"
At end messages were:
Python function terminated unexpectedly
Erno 131 Permission Denied: 'calibre_db_restore_report.txt (error code: 1
Traceback {most recent calls last}
File "site.py", line 32, in main
File "site.py", line 109, in run-entry-point
File "site-package\calibre\librarycli.py" Line 1554 in main
File "site-package\calibre\librarycli.py" Line 1343, in command-restore-database
IOError: (Erno 13) Permission denied: 'calibre_db_restore_report.txt

In the end I had a very large number of books in the "Extra Titles" which I gather means they are in the library but not in database (approc 8,500 books/magazines.pdf files etc)
I have 'Extra Authors" - around 1,400 in addition to above which are in library but not in database.
"Extra book format approx 200 in library but not database

So rebuild (if I did it correctly) did not really work.
Also have a large number of duplicate books over 8,000!!

Something has certainly gone wrong

mitch13

kovidgoyal · 02-12-2015, 01:16 AM

Rebuild will only work with extra titles if the file and folder names are in the correct format and they contain the correct opf file.

If you have random files and folders, then your best bet is to create a new library and do an add books from sub-folders, one book per folder.

As for that permission denied error, that will be because you ran calibredb restore in a folder for which you did not have permissions to write files (maybe the calibre progam folder?) All it means that the restore report could not be written.

mitch13 · 02-12-2015, 01:24 AM

Thanks Kovid.
Is there any way that books missing from the database can be created in the database by say editing metadata?
I ran the rebuild in my C:\ root directory.

mitch13

kovidgoyal · 02-12-2015, 01:41 AM

No, you have to re-add. Editing metadata will break things further.

mitch13 · 02-13-2015, 09:41 PM

Thanks for that Kovid and others.

Two final questions before I start an incredible amount of recovery.
What is the guaranteed method of Calibre Library backup so that the db files are OK?

Is there anyway now (or possible addon request) that Calibre can highlight books that are not in the database while viewing the Calibre library normally ie not having to run a report?

Cheers

mitch13

kovidgoyal · 02-13-2015, 10:04 PM

Just copy the calibre library folder. That is a perfectly good method of backup, the only caveat being that you should not be running calibre when doing the backup. Or if you wish to do it within calibre, you can use Save to Disk, but this will be much slower, as it updates metadata in all copies of the books.

mitch13 · 02-14-2015, 12:01 AM

Hi Kovid,
Is it possible to make a suggestion for an addition to Calibre. Faced with the job of rebuilding Calibre after my problems in mainly having items in the Library but not in the database. Rebuild the database did not help at all - reasons unknown and left me with Éxtra Titles' ie books not in db exceeding 10,000

Is there anyway now (or possible addon request) that Calibre can highlight books that are not in the database while viewing the Calibre library normally ie not having to run a report?

From the current report it is not easy to identify which books are not in the db especially if there are a lot of duplicates. Sure, I can run 'Find Duplicates' but then which one should I delete!

Is it possible the flag books that are in the library but not in the db while in the normal GUI?

Regards

mitch13

kovidgoyal · 02-14-2015, 12:08 AM

No, that would be a big performance hit. It implies having to read the entire list of files in the library folder and comparing it with the db. That can only be done in a check library type of job.

BookJunkieLI · 02-14-2015, 12:24 PM

An easy method for separating out which books are not in the database but are physically listed in the file structure is to use the Library=>Switch/Create Library=>Move the current library to a newly specified location option.

This will create a new library and move all of the books currently listed in your database to a new location. The books remaining in the old location are the ones that were not listed in the database. You can then work on adding them back into your library and, as long as you do not have Automerge selected in the Preferences=>Add Books section, it will notify you if you are trying to add any duplicates.

02-11-2015, 12:12 AM	#1
mitch13 Groupie Posts: 198 Karma: 10 Join Date: Jul 2008 Location: Queensland, Australia Device: Appraising the market	Calibre Database rebuild I have a very large library (in the tens of thousands). All has been going well for a number of years and Calibre performance on this large file is excellent. While working on my library recently (wanting to transfer all pdf files to a separate library) my PC had a problem and when it had finally recovered from its 'fit' had lost my entire calibre library. I did have a total backup even though it was a couple of days old and so I copied this library folder to the folder that had held my current library (now non-existent). Calibre then opened with close to the number of books I had. On running a library check it pronounced the library healthy but when followed by the next stage check I had a large number of books physically in the library but not in the database. After awhile and getting very nervous I decided to run a Restore Database believing that this would solve the problem. I have tried many times now and always during the second of three stages my PC goes into my least favourite mode ie 'Not Responding'. I then have to kill Calibre via Task Manager. Debug message: calibre Debug log calibre 2.19 [64bit] isfrozen: True is64bit: True Windows-8-6.2.9200 Windows ('64bit', 'WindowsPE') ('Windows', '8', '6.2.9200') Python 2.7.9 Windows: ('8', '6.2.9200', '', 'Multiprocessor Free') Starting up... Problem creating accessible interface for: CalibreSplashScreen(0x332b3f0) Make sure to deploy Qt with accessibility plugins. Problem creating accessible interface for: QScrollBar(0x87f5d80) Make sure to deploy Qt with accessibility plugins. Started up in 12.60 seconds with 0 books QTextCursor::setPosition: Position '1' out of range QTextCursor::setPosition: Position '382' out of range QTextCursor::setPosition: Position '87' out of range Any suggestions to recover my library built over about 4 years! Regards mitch13

02-11-2015, 03:02 AM	#4
mitch13 Groupie Posts: 198 Karma: 10 Join Date: Jul 2008 Location: Queensland, Australia Device: Appraising the market	Thanks Kovid. One dumb question - where do I run the command line from? Last edited by mitch13; 02-11-2015 at 03:43 AM.

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Database Rebuild... lost books	madelonw	Library Management	11	09-01-2014 08:22 PM
Database rebuild	Jellby	Kobo Reader	17	08-31-2014 04:25 AM
How to Rebuild Database	apswartz	Calibre	12	03-28-2010 01:11 PM
How to rebuild database in metafile	gandor62	Calibre	3	03-27-2010 08:31 PM
Force Database Rebuild?	davidm_uk	Calibre	2	01-13-2010 02:59 PM

02-11-2015, 12:13 AM	#2
JSWolf Resident Curmudgeon Posts: 78,907 Karma: 143095300 Join Date: Nov 2006 Location: Roslindale, Massachusetts Device: Kobo Libra 2, Kobo Aura H2O, PRS-650, PRS-T1, nook STR, PW3	No backup?

02-11-2015, 12:18 AM	#3
kovidgoyal creator of calibre Posts: 45,144 Karma: 27110892 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Just let it run, not responding just means the GUI is frozen not that the actual restore process has failed. This can happen if some restore operation is taking very long, blocking the GUI from responding until it completes. You can also run the restore from the command line, which removes the possibility of GUI problems. calibredb restore_database

02-11-2015, 03:12 AM	#5
kovidgoyal creator of calibre Posts: 45,144 Karma: 27110892 Join Date: Oct 2006 Location: Mumbai, India Device: Various	The windows command prompt (cmd.exe)

02-11-2015, 07:40 AM	#6
drjenkins Addict Posts: 250 Karma: 1702156 Join Date: Nov 2010 Device: Kindle Voyage	Sometimes when a GUI freezes I pull up Task Manager to get some reassurance that the process is still consuming CPU cycles.

02-12-2015, 12:20 AM	#7
mitch13 Groupie Posts: 198 Karma: 10 Join Date: Jul 2008 Location: Queensland, Australia Device: Appraising the market	Hi Kovid, Ran the Command version (DOS) as suggested - took around 20 hours but that not a problem. The command I used was c\: calibredb restore_database "Really-Do-It" At end messages were: Python function terminated unexpectedly Erno 131 Permission Denied: 'calibre_db_restore_report.txt (error code: 1 Traceback {most recent calls last} File "site.py", line 32, in main File "site.py", line 109, in run-entry-point File "site-package\calibre\librarycli.py" Line 1554 in main File "site-package\calibre\librarycli.py" Line 1343, in command-restore-database IOError: (Erno 13) Permission denied: 'calibre_db_restore_report.txt In the end I had a very large number of books in the "Extra Titles" which I gather means they are in the library but not in database (approc 8,500 books/magazines.pdf files etc) I have 'Extra Authors" - around 1,400 in addition to above which are in library but not in database. "Extra book format approx 200 in library but not database So rebuild (if I did it correctly) did not really work. Also have a large number of duplicate books over 8,000!! Something has certainly gone wrong mitch13

02-12-2015, 01:16 AM	#8
kovidgoyal creator of calibre Posts: 45,144 Karma: 27110892 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Rebuild will only work with extra titles if the file and folder names are in the correct format and they contain the correct opf file. If you have random files and folders, then your best bet is to create a new library and do an add books from sub-folders, one book per folder. As for that permission denied error, that will be because you ran calibredb restore in a folder for which you did not have permissions to write files (maybe the calibre progam folder?) All it means that the restore report could not be written.

02-12-2015, 01:24 AM	#9
mitch13 Groupie Posts: 198 Karma: 10 Join Date: Jul 2008 Location: Queensland, Australia Device: Appraising the market	Thanks Kovid. Is there any way that books missing from the database can be created in the database by say editing metadata? I ran the rebuild in my C:\ root directory. mitch13

02-12-2015, 01:41 AM	#10
kovidgoyal creator of calibre Posts: 45,144 Karma: 27110892 Join Date: Oct 2006 Location: Mumbai, India Device: Various	No, you have to re-add. Editing metadata will break things further.

02-13-2015, 09:41 PM	#11
mitch13 Groupie Posts: 198 Karma: 10 Join Date: Jul 2008 Location: Queensland, Australia Device: Appraising the market	Thanks for that Kovid and others. Two final questions before I start an incredible amount of recovery. What is the guaranteed method of Calibre Library backup so that the db files are OK? Is there anyway now (or possible addon request) that Calibre can highlight books that are not in the database while viewing the Calibre library normally ie not having to run a report? Cheers mitch13

02-13-2015, 10:04 PM	#12
kovidgoyal creator of calibre Posts: 45,144 Karma: 27110892 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Just copy the calibre library folder. That is a perfectly good method of backup, the only caveat being that you should not be running calibre when doing the backup. Or if you wish to do it within calibre, you can use Save to Disk, but this will be much slower, as it updates metadata in all copies of the books.

Advert

Advert

02-14-2015, 12:01 AM	#13
mitch13 Groupie Posts: 198 Karma: 10 Join Date: Jul 2008 Location: Queensland, Australia Device: Appraising the market	Hi Kovid, Is it possible to make a suggestion for an addition to Calibre. Faced with the job of rebuilding Calibre after my problems in mainly having items in the Library but not in the database. Rebuild the database did not help at all - reasons unknown and left me with Éxtra Titles' ie books not in db exceeding 10,000 Is there anyway now (or possible addon request) that Calibre can highlight books that are not in the database while viewing the Calibre library normally ie not having to run a report? From the current report it is not easy to identify which books are not in the db especially if there are a lot of duplicates. Sure, I can run 'Find Duplicates' but then which one should I delete! Is it possible the flag books that are in the library but not in the db while in the normal GUI? Regards mitch13

02-14-2015, 12:08 AM	#14
kovidgoyal creator of calibre Posts: 45,144 Karma: 27110892 Join Date: Oct 2006 Location: Mumbai, India Device: Various	No, that would be a big performance hit. It implies having to read the entire list of files in the library folder and comparing it with the db. That can only be done in a check library type of job.

02-14-2015, 12:24 PM	#15
BookJunkieLI Evangelist Posts: 435 Karma: 572984 Join Date: Jan 2010 Location: Long Island Device: Kobo Libra 2, Kindle 4, Nook Gl4, Nook STR, REB 1100, Ebookwise 1500,	An easy method for separating out which books are not in the database but are physically listed in the file structure is to use the Library=>Switch/Create Library=>Move the current library to a newly specified location option. This will create a new library and move all of the books currently listed in your database to a new location. The books remaining in the old location are the ones that were not listed in the database. You can then work on adding them back into your library and, as long as you do not have Automerge selected in the Preferences=>Add Books section, it will notify you if you are trying to add any duplicates.