Library crash - unable to rebuild

[Ellachanted]

Originally Posted by theducks

The GUI is updated at the end of A task.
Add FolderS is 1 task (100% when done)

Thank you

[LadyKate]

Hope you have success in salvaging at least part of your library Ellachanted.

I'm starting to wonder if we need to ask kovidgoyal to add a user feature "When to backup" that we set. Something we could set as number of hours or number of days from last backup (or off) that would have a message displayed (maybe starting to flash after it's ignored too long) that it is time to backup.

I've lost count of the number of people who have lost hours/weeks/months of work because they had no current backup when their library became corrupted. This is assuming that their backup has been checked and if in good form and working!

One other thing I have noticed is that most users post here after trying many things, some of which render other possibilities impossible. Perhaps we are in need of a sticky of the steps to take when confronted with library failure messages. Something people would be able to read when their library fails before they start deleting thing, re-adding their library from the library directory etc.

[eschwartz]

Why don't you write out the steps, and someone can sticky it.

[LadyKate]

Originally Posted by eschwartz

Why don't you write out the steps, and someone can sticky it.

I'll give it a try later this week. My problem is my first step would be make sure you have a working backup

ummm second step, don't delete anything, add anything or make any other changes until you have rebooted, then run a library repair. ...

Hummm I'm going to have to go through the old threads and see about this.

[MiniMouse]

Originally Posted by LadyKate

I'll give it a try later this week. My problem is my first step would be make sure you have a working backup

ummm second step, don't delete anything, add anything or make any other changes until you have rebooted, then run a library repair. ...

Hummm I'm going to have to go through the old threads and see about this.

I've wrote a report on my personal expirience with restoring after my library crash and there are also some replys that helped me a lot.

Though sadly I think we only know what really to do (and what works for oneself) after we went through rebuilding.

[cybmole]

there's 2 routes.
route 1 is you create a new, empty library then use add books to scan and bring in as many books as possible from the old location. This is very simple but you maybe lose some metadata.

route 2 is to try to fix the original library using the library repair tools within calibre

route 1 is non destructive of the original so could always be done anyway, to be safe
for route 2, it makes sense to take a complete image of the library folder before starting, that way you can't easily make things any worse, as you can restore to that image whenever things go pear shaped, & start over

I am not sure how much extra deatail is really needed

so i'ts make a backup.
do route 1.
try route 2

keep which ever is the best outcome

[Ellachanted]

I think backup timing depends on how often you add stuff. I was actually backing up every time I added things for a long time. But then life got in the way. And I'd forget. So a reminder might be good. My backup was about a month old. So it shouldn't have been too bad except for all those darn author updates. I love using Calibre for my fanfiction hoard. Because that is what it is. :P But trying to keep track of the latest version and add it to the back up library? I'm not always sure it takes. So my backup will always be questionable anyway.

I did seem to get the majority of the library back. Most of the metadata returned. There are more books than before though, so I'm not sure what got added that wasn't there before. Probably duplicates that should have been deleted.

At some point, when I am no longer spending most of my free time studying, I will have to combine all my libraries and clean them all up and create one perfect library that I can mess up again and a backup. Or three.

[eschwartz]

Well, that is the advantage of snapshotting backups -- you can set them to automatic, and if nothing changes for a while it won't take up any extra space. (Well, a megabyte or two just to store all the extra hardlink nodes.)

[Ellachanted]

Originally Posted by eschwartz

Well, that is the advantage of snapshotting backups -- you can set them to automatic, and if nothing changes for a while it won't take up any extra space. (Well, a megabyte or two just to store all the extra hardlink nodes.)

What is a "snapshot"? Is it the csv of the library?

I should do that more often.

[Ellachanted]

So after working for a week or so, the new library has crashed. Fortunately, it is mostly backed up. I am currently copying it to an external flash drive. I've also ordered a new Hard Drive. Because, frankly, this one showed errors when I ran disc repair. And it is a few years old and used a lot.

Hmm.. It seems that fixing the drive and installing the newest version of Calibre fixed things. So a new hard drive may stop these problems.

[HarryT]

Originally Posted by Ellachanted

So after working for a week or so, the new library has crashed. Fortunately, it is mostly backed up. I am currently copying it to an external flash drive. I've also ordered a new Hard Drive. Because, frankly, this one showed errors when I ran disc repair. And it is a few years old and used a lot.

Hmm.. It seems that fixing the drive and installing the newest version of Calibre fixed things. So a new hard drive may stop these problems.

If a drive starts showing errors on a scan, it's a sure sign that it's about to start pining for the fjords. You've done exactly the right thing in replacing the drive. It's a very, VERY good idea, if you're not already doing so, to use some form of sync software to keep your main library constantly backed up to an external drive. I use Microsoft Synctoy (a free download from Microsoft) but there are several other similar tools available.

[cybmole]

something like a large fan fiction library could be v hard to rebuild from scratch - so put an extra copy into cloud storage also, to be extra safe,

just pick one ( or more) of google drive, dropbox, one drive, amazon cloud drive... they all give you a few Gb for free.... and copy your entire calibre folder to there ( then don't forget the password!


then even if your entire PC goes up in flames, you still have that library !

[eschwartz]

Originally Posted by Ellachanted

What is a "snapshot"? Is it the csv of the library?

I should do that more often.

A snapshot is an image of the entire library at a given time. I use the linux/unix tool "rsync", setup similar to Apple's Time Machine. But I believe you can do the same with Windows programs like MS SyncToy, FreeFileSync, and others.

The idea is: you create a full backup, once. Then the next time you backup, you link the backup to the first backup, and so on.
Each backup is an independent, full backup, ideally marked with the date it was made , but using hardlinks between the backups to conserve space if a file didn't change in that time.

Set up like this, you only use up space if files have changed, which means you can use a system task (cron/systemd on linux, launchd on OSX, Scheduled Tasks on Windows) to keep backups for every day without it taking over your whole hard drive. Depending on the program, it might be able to set a task itself.


Set it once and forget about it.

[BookJunkieLI]

Originally Posted by cybmole

something like a large fan fiction library could be v hard to rebuild from scratch - so put an extra copy into cloud storage also, to be extra safe,

just pick one ( or more) of google drive, dropbox, one drive, amazon cloud drive... they all give you a few Gb for free.... and copy your entire calibre folder to there ( then don't forget the password!

This is very very true. Sites disappear or authors take their work down, sometimes within days of putting it up. (Which I don't get but it's their right.) And you can't count on the Wayback Machine to have copies. Always keep an extra backup copy of your library.

[LadyKate]

Originally Posted by Ellachanted

I think backup timing depends on how often you add stuff. I was actually backing up every time I added things for a long time. But then life got in the way. And I'd forget. So a reminder might be good. My backup was about a month old. So it shouldn't have been too bad except for all those darn author updates. I love using Calibre for my fanfiction hoard. Because that is what it is. :P But trying to keep track of the latest version and add it to the back up library? I'm not always sure it takes. So my backup will always be questionable anyway.

I did seem to get the majority of the library back. Most of the metadata returned. There are more books than before though, so I'm not sure what got added that wasn't there before. Probably duplicates that should have been deleted.

At some point, when I am no longer spending most of my free time studying, I will have to combine all my libraries and clean them all up and create one perfect library that I can mess up again and a backup. Or three.

On advice from here I tried FreeFileSync and now do a quick backup to a mirror directory on another drive (actually a usb drive that is always connected). As it is an incremental backup it doesn't take much time but as it backs up some files that are usually in use by calibre I run it after exiting. It takes a bit of time to scan the directories if my system has been powered down since the last update but does not need my intervention to run.

It has become a habit to run it almost every day even if I've only tweeked the metadata on one or two books.