A lesson on the importance of backups...

Originally Posted by HarryT

Strange. I can't think of any reason it wouldn't copy a file unless some other program had the file open.

That will be a common problem if you're backing up the entire user folder, mostly due to the AppData folders. The problem also occurs frequently with documents if you leave your applications open (e.g. during scheduled backups). Many backup applications can get around that, as the backup process would be fairly unreliable otherwise.

[Fbone]

Originally Posted by HarryT

Strange. I can't think of any reason it wouldn't copy a file unless some other program had the file open.

Oh well, it works for me, and that's all that matters to me .

I remember one of the files was Calibre's metadata.db file. I don't know what the file does but I had to manually copy each one every time I synced.

[crich70]

Originally Posted by Fbone

I remember one of the files was Calibre's metadata.db file. I don't know what the file does but I had to manually copy each one every time I synced.

Offhand I'd guess that it has something to do with keeping the things like Author's name and Book Title sorted in Calibre. db = database in my mind.

Originally Posted by crich70

Offhand I'd guess that it has something to do with keeping the things like Author's name and Book Title sorted in Calibre. db = database in my mind.

It is an SQLite database, and it stores exactly what you said (plus a lot more).

[wawasteele]

Originally Posted by Fbone

I remember one of the files was Calibre's metadata.db file. I don't know what the file does but I had to manually copy each one every time I synced.

I use SyncToy and that happens if I forget to close Calibre first. If Calibre is closed, I've never had that problem.

[AnotherCat]

I used SyncToy from its beginning through to sometime after the last (now very dated) version. In more recent times I have used FreeFileSync which is very robust and in my view much easier to use and set up.

SyncToy suffered from a number of problems and as far as I know, for example, errors and redundant backing up of files due to daylight savings time changes were never fixed nor ever formally described (SyncToy was never a formal Microsoft product and so was never supported). I would be very wary using it especially if syncing both NTFS and FAT Drives as NTFS time is UT format and FAT is local time (FreeFileSync, for example, specifically provides for sync'ing these).

FreeFileSync is rigorous in its handling of such issues, others use different syncing applications and I suspect they may be just as good and maybe better but I have had no need to try them.

[HarryT]

Originally Posted by Fbone

I remember one of the files was Calibre's metadata.db file. I don't know what the file does but I had to manually copy each one every time I synced.

Sounds as if you had Calibre running. You need to make sure you close it before you do the backup.

[Freeshadow]

http://ipmsg.org/tools/fastcopy.html.en

Offers diff, copy and sync modes and runs without making use of windows' own copy routines = higher limits on path lengths.

Has both CLI and GUI interfaces.

Have a look/read/try

[Fbone]

Originally Posted by HarryT

Sounds as if you had Calibre running. You need to make sure you close it before you do the backup.

Apparently. The program I use now has no such requirement.

[HarryT]

Originally Posted by Fbone

Apparently. The program I use now has no such requirement.

The program you're using now is behaving in an unsafe manner; it's evidently simply getting a read lock, and not an exclusive lock, on the files it's trying to copy, and when backing up a SQL database such as the one that Calibre uses this is dangerous, because you could well be backing up the file with potentially uncommitted data. This is the same issue that causes people to end up with a corrupted Calibre database if they put Calibre on a network drive.

SyncToy is behaving absolutely correctly in telling you "I can't back up this file - some other application has opened it.". You really should heed its warnings, not regard them as a problem. In all honesty, I would not trust whatever backup tool you're now using if it's behaving in the manner that you describe. Your backed-up data may very well be corrupt.

Trust me - I do this stuff for a living.

[JSWolf]

Originally Posted by HarryT

The program you're using now is behaving in an unsafe manner; it's evidently simply getting a read lock, and not an exclusive lock, on the files it's trying to copy, and when backing up a SQL database such as the one that Calibre uses this is dangerous, because you could well be backing up the file with potentially uncommitted data. This is the same issue that causes people to end up with a corrupted Calibre database if they put Calibre on a network drive.

I just make sure I am not running multiple copies of Calibre on the same library. Network issues aren't issues if you do that.

[Fbone]

Originally Posted by HarryT

The program you're using now is behaving in an unsafe manner;

/edited/

Trust me - I do this stuff for a living.

Thanks, Harry for the warning. I'll try to remember to close the program first. Since I back-up daily, closing (and restarting) Calibre and all the rest is a nuisance that I hoped to avoid.

[shalym]

Originally Posted by HarryT

The program you're using now is behaving in an unsafe manner; it's evidently simply getting a read lock, and not an exclusive lock, on the files it's trying to copy, and when backing up a SQL database such as the one that Calibre uses this is dangerous, because you could well be backing up the file with potentially uncommitted data. This is the same issue that causes people to end up with a corrupted Calibre database if they put Calibre on a network drive.

SyncToy is behaving absolutely correctly in telling you "I can't back up this file - some other application has opened it.". You really should heed its warnings, not regard them as a problem. In all honesty, I would not trust whatever backup tool you're now using if it's behaving in the manner that you describe. Your backed-up data may very well be corrupt.

Trust me - I do this stuff for a living.

I also "do this stuff for a living", and in the real world, most databases are NOT shut down every night so that a backup can be done. Most *modern* backup programs can and do handle database backups without completely shutting down the program and without corruption. I maintain several MS SQL and MySQL databases that are backed up every night, with test restores done at least monthly, and I can tell you that I've never had a backup corrupted because a database was open while it was being backed up.

Shari

[HarryT]

Originally Posted by shalym

I also "do this stuff for a living", and in the real world, most databases are NOT shut down every night so that a backup can be done. Most *modern* backup programs can and do handle database backups without completely shutting down the program and without corruption. I maintain several MS SQL and MySQL databases that are backed up every night, with test restores done at least monthly, and I can tell you that I've never had a backup corrupted because a database was open while it was being backed up.

Shari

But you presumably do your database backups using the backup facilities of MS SQL Server itself (just as I do), not by simply copying the underlying file system files. If you use the backup facility of the database itself, then yes, it does it properly, no argument about that, and no need to shut anything down. But that's not what we're talking about in this situation: it is assuredly not safe to simply copy the files of an open database. You might get away with it 9 times out of 10, or even 99 times out of a hundred, but sooner or later you'll hit problems when you have pending transactions that haven't been committed to the database.

[Katsunami]

Originally Posted by shalym

I also "do this stuff for a living", and in the real world, most databases are NOT shut down every night so that a backup can be done. Most *modern* backup programs can and do handle database backups without completely shutting down the program and without corruption. I maintain several MS SQL and MySQL databases that are backed up every night, with test restores done at least monthly, and I can tell you that I've never had a backup corrupted because a database was open while it was being backed up.

Calibre uses SQLite. That is a relational database stored in a simple file. It doesn't use a server. Therefore, when backing up, normal file backup procedures (and pitfalls) apply. Maybe one day Kovid could port the database over to MySQL or something else (Maria, Postgre), and so integrate a portable, auto-starting server. Calibre would then be running a "proper" database, but the distribution would become about twice as big and the total program would become even heavier. While I love Calibre for what it does, I can't say it's a light program to run.