Library Blank - rebuit DB but not the same

JustinD · 03-20-2016, 07:09 PM

So the other day I noticed my Calibre library was blank. No matter what I did I couldn't see any of the underlying books. I found in the documentation regarding a corrupt db so I had Calibre rebuild the library. After 4 days of chugging (~13k ebooks) I had my library back.

But, I had constructed some more metadata such as a field for when read (including date) and a field for "to read". I had also some virtual libraries where I had books for my kids age groups. All of this is no longer there

.

Does anyone know what I need to do to get this back or is it the case when the db is rebuilt it goes back to stock db fields? Is there any way to either get them back now or protect against a possible future issue so I don't lose them?

Justin

eschwartz · 03-20-2016, 07:18 PM

Custom columns (and any other preferences set in the database rather than in the config folder) can be restored from the "metadata_db_prefs_backup.json" file that sits next to the metadata.db

It sounds like the prefs backup was broken or missing.

This is one reason why restoring the database does not always fix everything -- although it certainly tries to recover as much information as it can!

...

Do you have any backups?

JustinD · 03-20-2016, 07:47 PM

I run 90 day backups and last it was run mid December 2015. I have added books since then so that why I just let the db attempt to rebuild itself.

Is there any way to take the old (Dec) pref backup and apply it now?

eschwartz · 03-20-2016, 08:50 PM

Well, the column definitions are in the pref backup, and restoring with that *should* restore the columns. Simply copy the backup prefs in place and rerun the restore.
(Alternatively, you can recreate the columns from memory -- they might match. Or analyze the metadata.opf stored definitions, though that is kinda complicated.)

But the per-book values are of course in the individual metadata.opf files -- and it depends whether calibre has rewritten those based on the current database yet.

calibre writes a backup of the current metadata whenever a book record changes -- which is good, unless the current metadata is wrong and the backup is right!

A restore *shouldn't* touch the metadata.opf backups, IIRC.
Opening calibre after the restore and modifying a book record will, at least for that book record.

So if I were restoring my books, I would back them up first, until I knew everything was in order. This is important -- don't allow any of the pieces you currently have to become destroyed, until you know you don't need it anymore!

You might need to play around with the columns to match what the metadata.opf thinks should be there, but in theory the data you want is not quite gone yet and can still be recovered.

BetterRed · 03-20-2016, 09:10 PM

Quote:

Originally Posted by JustinD

I run 90 day backups and last it was run mid December 2015. I have added books since then so that why I just let the db attempt to rebuild itself.

Is there any way to take the old (Dec) pref backup and apply it now?

Sort of what he ↑↑↑↑ said.

Not sure if it will do you any good but you could rename the one that's there to something like save_[...]_backup.json then copy the one from back up and do another restore and see what happens.

But its probably safer and faster to find the books you've added since the backup - use the Date Search tab in "Advanced Search" - binoculars to left of search bar - on the Date column, by default it contains the time stamp of when the book was added. Save those books to Disk somewhere.

Then restore the backup, preferably not over the 'broken' library, and then add the Books you just Saved to Disk to the restored library. Once your satisfied you have a working library you can delete the 'broken' one.

Then I suggest you find an incremental backup solution that suits your Operating and File Systems, hopefully that will encourage more frequent backups - daily even

BR

JustinD · 03-20-2016, 09:13 PM

thanks. I will look at my backup tonight and grab the "metadata_db_prefs_backup.json" file. And then restore my restored db!
I will report back in 4-5 days (given how long it took last time)

Justin

JustinD · 04-01-2016, 10:38 PM

Hi all to those that kindly gave me suggestions to fully recover my library. I did another restore using the 3 month backup version of the metadata_db_prefs file and it worked a treat! Now that I know I can somehow get data base corruption I have changed my backups to monthly with 3 historical versions (since the total library size isn't that large). My backups happen automatically so I shouldn't have issue again. Thanks both eschwartz and BetterRed

eschwartz · 04-02-2016, 09:51 PM

Happy to help -- and thanks for reporting on your successful resolution!

03-20-2016, 07:09 PM	#1
JustinD Member Posts: 16 Karma: 10 Join Date: Mar 2011 Device: iPad mini, iPhone, Kindle	Library Blank - rebuit DB but not the same So the other day I noticed my Calibre library was blank. No matter what I did I couldn't see any of the underlying books. I found in the documentation regarding a corrupt db so I had Calibre rebuild the library. After 4 days of chugging (~13k ebooks) I had my library back. But, I had constructed some more metadata such as a field for when read (including date) and a field for "to read". I had also some virtual libraries where I had books for my kids age groups. All of this is no longer there . Does anyone know what I need to do to get this back or is it the case when the db is rebuilt it goes back to stock db fields? Is there any way to either get them back now or protect against a possible future issue so I don't lose them? Justin

03-20-2016, 07:47 PM	#3
JustinD Member Posts: 16 Karma: 10 Join Date: Mar 2011 Device: iPad mini, iPhone, Kindle	I run 90 day backups and last it was run mid December 2015. I have added books since then so that why I just let the db attempt to rebuild itself. Is there any way to take the old (Dec) pref backup and apply it now? Last edited by JustinD; 03-20-2016 at 08:38 PM.

03-20-2016, 08:50 PM	#4
eschwartz Ex-Helpdesk Junkie Posts: 19,422 Karma: 85397180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	Well, the column definitions are in the pref backup, and restoring with that should restore the columns. Simply copy the backup prefs in place and rerun the restore. (Alternatively, you can recreate the columns from memory -- they might match. Or analyze the metadata.opf stored definitions, though that is kinda complicated.) But the per-book values are of course in the individual metadata.opf files -- and it depends whether calibre has rewritten those based on the current database yet. calibre writes a backup of the current metadata whenever a book record changes -- which is good, unless the current metadata is wrong and the backup is right! A restore shouldn't touch the metadata.opf backups, IIRC. Opening calibre after the restore and modifying a book record will, at least for that book record. So if I were restoring my books, I would back them up first, until I knew everything was in order. This is important -- don't allow any of the pieces you currently have to become destroyed, until you know you don't need it anymore! You might need to play around with the columns to match what the metadata.opf thinks should be there, but in theory the data you want is not quite gone yet and can still be recovered. Last edited by eschwartz; 03-20-2016 at 08:53 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Blank pages	Sablerose	Marvin	11	02-08-2014 09:06 AM
Blank Lines	jreidu	Workshop	2	07-20-2011 05:11 AM
Blank books	Mememememe	Kobo Reader	3	05-11-2010 04:33 PM
Blank Lines	vivaldirules	Upload Help	55	03-02-2009 03:17 PM
Blank pages	JSWolf	Calibre	4	11-29-2008 01:22 PM

03-20-2016, 07:18 PM	#2
eschwartz Ex-Helpdesk Junkie Posts: 19,422 Karma: 85397180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	Custom columns (and any other preferences set in the database rather than in the config folder) can be restored from the "metadata_db_prefs_backup.json" file that sits next to the metadata.db It sounds like the prefs backup was broken or missing. This is one reason why restoring the database does not always fix everything -- although it certainly tries to recover as much information as it can! ... Do you have any backups?

03-20-2016, 09:13 PM	#6
JustinD Member Posts: 16 Karma: 10 Join Date: Mar 2011 Device: iPad mini, iPhone, Kindle	thanks. I will look at my backup tonight and grab the "metadata_db_prefs_backup.json" file. And then restore my restored db! I will report back in 4-5 days (given how long it took last time) Justin

04-01-2016, 10:38 PM	#7
JustinD Member Posts: 16 Karma: 10 Join Date: Mar 2011 Device: iPad mini, iPhone, Kindle	Hi all to those that kindly gave me suggestions to fully recover my library. I did another restore using the 3 month backup version of the metadata_db_prefs file and it worked a treat! Now that I know I can somehow get data base corruption I have changed my backups to monthly with 3 historical versions (since the total library size isn't that large). My backups happen automatically so I shouldn't have issue again. Thanks both eschwartz and BetterRed

04-02-2016, 09:51 PM	#8
eschwartz Ex-Helpdesk Junkie Posts: 19,422 Karma: 85397180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	Happy to help -- and thanks for reporting on your successful resolution!

Advert

Advert