Personal experience with restoring library after crash

[MiniMouse]

Hi,

just felt the need to post my personal experience with restoring a crashed library because YES, that happened to me a few days ago! And maybe my experience is old news to most of you but for me it was new territory.

So, this is my story:
I worked a whole weekend on optimizing my fanfiction library - optimizing my settings, optimizing my custom columns, optimizing my templates, optimizing my catalog functions - just optimizing in general all that needed (in my opinion) to be optimized. And believe me when I say I found a lot!

Further I experimented with dropbox AND cleared my whole reader and played my whole fanfiction library back on.

The only thing missing was making a backup.

But as I wanted to do that Sunday evening... opened calibre SHOCK no stories in it! ALL 1.870 totally optimized stories GONE!!! ALL my settings, custom columns GONE!!!! My whole calibre library set back to the default settings with NOTHING in it!

I don't really know what happened but it did happen (though personally I blame dropbox for it)!

After my first heart-attack was over, my journey to get it back was on. And before you ask
- no, pointing calibre again to the location of the library didn't work
- no, dropbox was not syncing (and I only put a copy of my library up there anyway)
- yes, I had a backup but it was a month old AND before all my optimization
- and yes, the book folders (with the book files) were still in the library folder (though I found out later that 16 books were missing at that time)

So I learned a few things that are good to know... next time... should it ever happen again...
- to have a backup in a newest version is always good
- to have your complete library in a newest version on your reader is very good (if you don't have the first thing)
- to have a document file with all settings, custom columns, templates of your calibre is priceless (if you don't have the first two things)


You want to know why?

- When you try 'Library Maintenance → Restore database' you have to rebuilt ALL you custom columns and templates first, otherwise you will only get the default installed columns filled (because calibre can't write any data in columns that don't exist even when the data for them is stored in the books metadata). A point that wasn't made really clear for me in the calibre manual.
- Copy the metadata and metadata_prefs files from your library folder AFTER rebuilding all your settings and custom columns BEFORE starting the restore because then you don't have to make them every time new when it doesn't work the way you want it (made this mistake two times until I found that out ).
- And a problem that I had too, though I don't know if it really was a mistake on my part or the other way around. I tried three times but the restore function always skipped one or two custom columns. They were gone after the restore (and I don't think I made the same mistakes in my templates and naming of the columns three times). The fourth time I put the custom columns more or less in the same order than they were before my crash and then it worked.

As I connected my reader to the new restored library, I found out though that 16 books were missing!
And then I found out that I can also import my books from my reader to my library (though I don't know if every reader stores all the metadata information the same way than Sony reader do)! So I added the books to the library from my reader and SAVED that now complete-again library.

Then I tried the restore again, though this time NOT over the restore function from calibre. This time I tried it directly over my reader (because as I said before I had my complete and newest version of my library on it anyway). So I opened an empty library (with all my custom columns and settings) and imported the whole library directly from my reader. Worked also perfectly!

So, you see? Having a backup of ALL your settings and custom columns (and templates in them) is a very good thing, additional to an up to date library backup or an up to date reader backup.

Of course the first thing I did after I had everything back, was making a backup of everything. Just in case...

So, that was my personal experience with a crashed library, what was yours?

Mini

[chaley]

I am glad you managed to rebuild your library. That is certainly the important thing.

Thank you for posting. It is always useful to have clear and concise feedback on a process.

Originally Posted by MiniMouse

- When you try 'Library Maintenance → Restore database' you have to rebuilt ALL you custom columns and templates first, otherwise you will only get the default installed columns filled (because calibre can't write any data in columns that don't exist even when the data for them is stored in the books metadata). A point that wasn't made really clear for me in the calibre manual.

When restore is run, it checks for the existence of the file metadata_db_prefs_backup.json. If that file is there, it is assumed to contain the correct custom column definitions. If it is not there or if it is corrupted, restore will scan every book looking for metadata.opf files and rebuild the custom column information from what it finds.

To get the behavior you saw, three things must be true:
1) there was no metadata.db file in your library folder, and
2) calibre successfully initialized a database in that folder, and
3) calibre successfully closed, which is when metadata_db_prefs_backup.json is written.

This sequence of steps will overwrite the metadata_db_prefs_backup.json file with initial values, preventing the restore of all the custom column information. You can force calibre to do the scan by deleting the metadata_db_prefs_backup.json file then running the restore without closing calibre (from the command line would be good). Of course, you wouldn't know to do this.

One does wonder why your library metadata.db disappeared ...

One reason that metadata_db_prefs_backup.json exists is that custom column changes take time to appear in all the metadata.opf files. Until all the files are brought up to date, the scan will produce conflicting information. Calibre attempts to handle those conflicts by assuming that the newest metadata.opf contains the correct information, but that might not be true if no metadata.opf files were written or if the file is corrupted.

I wonder if I shouldn't change calibre to do both the scan and the load from metadata_db_prefs_backup.json. It isn't clear what to do with the results. My feeling is:
a) a column definition (a lookup key) found in metadata_db_prefs_backup.json should win over the same definition found in a metadata.opf, and
b) a column definition found in the scan that is not in metadata_db_prefs_backup.json should be added to the db.

This scheme could re-add old custom columns to the db, but that is easily fixed once the recovery finished. It would also deal with the problem you must have had, where the metadata_db_prefs_backup.json file was reinitialized somehow before the recovery was run.

[MiniMouse]

Originally Posted by chaley

I wonder if I shouldn't change calibre to do both the scan and the load from metadata_db_prefs_backup.json. It isn't clear what to do with the results. My feeling is:
a) a column definition (a lookup key) found in metadata_db_prefs_backup.json should win over the same definition found in a metadata.opf, and
b) a column definition found in the scan that is not in metadata_db_prefs_backup.json should be added to the db.

This scheme could re-add old custom columns to the db, but that is easily fixed once the recovery finished. It would also deal with the problem you must have had, where the metadata_db_prefs_backup.json file was reinitialized somehow before the recovery was run.

First I have to say, I'm really happy that there IS a restore function!

Normally for people like me a crash is something new and you do what the manual is suggesting (you don't really know about what info is stored where and what will happen). So I think restoring too much columns would definitely be a lesser problem than restoring only the default columns.

And maybe there is a chance to save ALL custom columns (with templates and all). Like a back door, a file you can save somewhere else and load back in. Or is this function already there and I don't know about it?

Mini

[eschwartz]

Originally Posted by MiniMouse

And maybe there is a chance to save ALL custom columns (with templates and all). Like a back door, a file you can save somewhere else and load back in. Or is this function already there and I don't know about it?

Mini

Yes, just save the metadata_db_prefs_backup.json file.

[chaley]

Originally Posted by MiniMouse

And maybe there is a chance to save ALL custom columns (with templates and all). Like a back door, a file you can save somewhere else and load back in. Or is this function already there and I don't know about it?

Another way to do it is to create a new library with settings from the current library. Don't put books in it. Now you have a library with all the settings and templates that you can use to (re)create other libraries at need.

[MiniMouse]

Originally Posted by eschwartz

Yes, just save the metadata_db_prefs_backup.json file.

Good to know, didn't think about that before.

Originally Posted by chaley

Another way to do it is to create a new library with settings from the current library. Don't put books in it. Now you have a library with all the settings and templates that you can use to (re)create other libraries at need.

Yes, did that. But then you have to make always a new library or delete the metadata and metadata_prefs first to copy your settings in your backup library. Thought there might be a better way.

Mini

[eschwartz]

Dropbox is a handy way of syncing this, including your whole library too!
(Just don't leave it syncing while calibre is actually running. )

You can also use command-line scripts with tools like rsync (see linux example here) which could be modified to use a windows batch file with rsync for Windows -- assuming you are techie enough to do so, I guess.

It is, of course, much easier to do on linux.

[BetterRed]

Originally Posted by MiniMouse

Yes, did that. But then you have to make always a new library or delete the metadata and metadata_prefs first to copy your settings in your backup library. Thought there might be a better way.

Mini

@MiniMouse - perhaps this command line can help ==>> calibredb — clone

BR

[cvlowe]

For a backup, I use Sync Toy once a day on my My Documents folder (Synchronize option) to a mirror on my external hard drive. My 'Library', 'Library.Personal' and 'Library.Computer' folders are all located in the root of my My Documents folder. I usually run it when I boot up the computer after work, before opening Calibre. Is this "safe"? Is there a chance of the library corrupting sometime after shut down or during boot up that I will then syncronize before opening Calibre? I also have an additional external drive I periodically swap out and always keep one stored in a fireproof safe. By periodically, I mean 3-6 months, I should probably make a point to swap more often...

[eschwartz]

Originally Posted by cvlowe

For a backup, I use Sync Toy once a day on my My Documents folder (Synchronize option) to a mirror on my external hard drive. My 'Library', 'Library.Personal' and 'Library.Computer' folders are all located in the root of my My Documents folder. I usually run it when I boot up the computer after work, before opening Calibre. Is this "safe"? Is there a chance of the library corrupting sometime after shut down or during boot up that I will then syncronize before opening Calibre? I also have an additional external drive I periodically swap out and always keep one stored in a fireproof safe. By periodically, I mean 3-6 months, I should probably make a point to swap more often...

The main thing is to never sync while calibre is running, like with any other sync solution such as Dropbox.

When the two programs are trying to perform operations on your folders at the same time, they may end up with a conflicting idea of what should be there, and either preserve what the other deleted, or delete (refuse to see) what the other created.

[MiniMouse]

Originally Posted by BetterRed

@MiniMouse - perhaps this command line can help ==>> calibredb — clone

BR

I have to check out how to use command line first (never have worked with it before). But I think that is exactly what I was looking for.

[Gary_M_Mugford]

Originally Posted by chaley

I wonder if I shouldn't change calibre to do both the scan and the load from metadata_db_prefs_backup.json. It isn't clear what to do with the results. My feeling is:
a) a column definition (a lookup key) found in metadata_db_prefs_backup.json should win over the same definition found in a metadata.opf, and
b) a column definition found in the scan that is not in metadata_db_prefs_backup.json should be added to the db.

This scheme could re-add old custom columns to the db, but that is easily fixed once the recovery finished. It would also deal with the problem you must have had, where the metadata_db_prefs_backup.json file was reinitialized somehow before the recovery was run.

Chaley, old friend,

Might I suggest that the start the rebuild with a question. "Do you want to choose when conflicts occur?" [N]

With a small library, the length of time of the scan would be short enough to afford the user the ability to sit and monitor the process. With bigger libraries, letting the computer decide, using the basis you have outlined here, would probably be appropriate. Instructions would need to make it crystal clear that rebuilding is not an instantaneous process. Been there, done that, got the eyestrain to show for it. But then again, my library is a BIT larger than normal.

One of the things I do is regularly run (and especially after some heavy editing/changing) a windows batch file. The concept should easily translate into linux or OSX.

Spoiler:

G:
cd \GaryELibQ

if exist metadata.db.7 del metadata.db.7
if exist metadata.db.6 ren metadata.db.6 metadata.db.7
if exist metadata.db.5 ren metadata.db.5 metadata.db.6
if exist metadata.db.4 ren metadata.db.4 metadata.db.5
if exist metadata.db.3 ren metadata.db.3 metadata.db.4
if exist metadata.db.2 ren metadata.db.2 metadata.db.3
if exist metadata.db.1 ren metadata.db.1 metadata.db.2
if exist metadata.db copy metadata.db metadata.db.1
if exist metadata.db copy metadata.db N:\_Backup_CalibreNonBookData

if exist metadata_db_prefs_backup.json.7 del metadata_db_prefs_backup.json.7
if exist metadata_db_prefs_backup.json.6 ren metadata_db_prefs_backup.json.6 metadata_db_prefs_backup.json.7
if exist metadata_db_prefs_backup.json.5 ren metadata_db_prefs_backup.json.5 metadata_db_prefs_backup.json.6
if exist metadata_db_prefs_backup.json.4 ren metadata_db_prefs_backup.json.4 metadata_db_prefs_backup.json.5
if exist metadata_db_prefs_backup.json.3 ren metadata_db_prefs_backup.json.3 metadata_db_prefs_backup.json.4
if exist metadata_db_prefs_backup.json.2 ren metadata_db_prefs_backup.json.2 metadata_db_prefs_backup.json.3
if exist metadata_db_prefs_backup.json.1 ren metadata_db_prefs_backup.json.1 metadata_db_prefs_backup.json.2
if exist metadata_db_prefs_backup.json copy metadata_db_prefs_backup.json metadata_db_prefs_backup.json.1
if exist metadata_db_prefs_backup.json copy metadata_db_prefs_backup.json N:\_Backup_CalibreNonBookData

if exist metadata_pre_restore.db.7 del metadata_pre_restore.db.7
if exist metadata_pre_restore.db.6 ren metadata_pre_restore.db.6 metadata_pre_restore.db.7
if exist metadata_pre_restore.db.5 ren metadata_pre_restore.db.5 metadata_pre_restore.db.6
if exist metadata_pre_restore.db.4 ren metadata_pre_restore.db.4 metadata_pre_restore.db.5
if exist metadata_pre_restore.db.3 ren metadata_pre_restore.db.3 metadata_pre_restore.db.4
if exist metadata_pre_restore.db.2 ren metadata_pre_restore.db.2 metadata_pre_restore.db.3
if exist metadata_pre_restore.db.1 ren metadata_pre_restore.db.1 metadata_pre_restore.db.2
if exist metadata_pre_restore.db copy metadata_pre_restore.db metadata_pre_restore.db.1
if exist metadata_pre_restore.db copy metadata_pre_restore.db N:\_Backup_CalibreNonBookData

N:
cd \_Backup_CalibreNonBookData
if exist CData.7 del CData.7
if exist CData.6 ren CData.6 CData.7
if exist CData.5 ren CData.5 CData.6
if exist CData.4 ren CData.4 CData.5
if exist CData.3 ren CData.3 CData.4
if exist CData.2 ren CData.2 CData.3
if exist CData.1 ren CData.1 CData.2
if exist CData copy CData CData.1
copy C:\Users\Gary\AppData\Roaming\calibre\*.* CData

The batch file might lead to a large set of backup files as the metadata.db files can reach the hundreds of megabytes. But the json file and the pre_restore database are much, much smaller, just kilobytes for the settings.json file. Also, admittedly, seven backups are a little much. You can set up, so that one of the backup sets, goes off-disk to another storage location, as I have done here. You might decide not to do that. Or you might decide to avail yourself of Dropbox or something similar.

Of course, the usual caveat. Don't run this batch file while Calibre is running. AND, make backups of your precious Calibre BOOK files to boot. Backups are never complete until you have a backup on another piece of hardware AND a copy in the clouds. So, when your computer has a metal (sic) breakdown, you can be sure the library will still be there. The device won't always have all books, although frequently it WILL pass for the first half of the backup process, the saving of the library itself to a off-computer piece of hardware.

PS: and if you are wondering how I got acquainted with rebuild when I have such a fine and all-encompassing backup process? One is the result of the other. [G]

[eschwartz]

I suspect you would get better mileage out of a versioned snapshot system similar to the example I mentioned in post #7

Rather than simply maintaining x versions and firing off a backup script to rotate it, this would create a Time Machine in which you can go back and restore any files you need. The main thing is though, that if nothing changed you do not keep multiple copies, but simply hardlink one copy to multiple locations.

Anyway, using timestamps (like the example) is not only more descriptive, it also reduces the needed commands to two. One to create a new backup, the other to delete all files older than x. (we assume you will notice the problems sometime within a month.)

[AlexBell]

Originally Posted by MiniMouse

Hi,

just felt the need to post my personal experience with restoring a crashed library because YES, that happened to me a few days ago! And maybe my experience is old news to most of you but for me it was new territory.

<snippped>

As I connected my reader to the new restored library, I found out though that 16 books were missing!
And then I found out that I can also import my books from my reader to my library (though I don't know if every reader stores all the metadata information the same way than Sony reader do)! So I added the books to the library from my reader and SAVED that now complete-again library.

<snipped>

Mini

Could you tell me please how you imported ebooks from your Sony reader?

I had a recent hard disk crash, and have lost all data from two months ago. NB: I know and use only the basics of calibre.

[davidfor]

Originally Posted by AlexBell

Could you tell me please how you imported ebooks from your Sony reader?

I had a recent hard disk crash, and have lost all data from two months ago. NB: I know and use only the basics of calibre.

You can do it in two ways. From within calibre, go to the device list, select the books, right click on them and choose "Add to library".

Alternatively, connect the device and use your file manager to copy the books from the device to the PC and calibre.