Backup Calibre desktop to Portable?

[mbovenka]

Originally Posted by BetterRed

However I recall seeing another file sync tool that sprinkled stuff in the directories selected for syncing. So if you mirrored D:\Calibre\Calibre Library to E:\Calibre\Calibre Library, it would write 'control' data into those directories which would generate errors in Check Library. I can't recall if FFS was the culprit or the solution.

The solution, I think. I use FFS to backup my Calibre setup, and I have no issues with Check Library. It puts a lock file in the source directory, but removes it when done.

[latepaul]

Here's what I would do:

• Plug the USB drive into the PC with the "master" library.
• open Calibre portable - sort the library by date (the date added column)
• make a note of the latest book and the date
• Use switch/create to switch to the master library by specifying the location on the PC
• sort by date, select all books newer than the portable library
• right-click and choose copy to library, choosing the calibre portable library

This assumes the PC copy is always the definitive copy. If you update the metadata on an existing book you could include it by always updating the date field. Or you could use a tag if you want the date field to always mean date added.

If you want to sync the other way - portable -> PC - then you need a way of identifying those changes. Again I'd probably use a tag.

The reason I'd would use this method is that it avoids manipulating files except through the calibre interface. Also, in most cases if you get into the habit of doing it regularly the changes will not be that large.

Oh and another tip - name the directories so that it's obvious what the libraries are ("usb_lib", "PC_lib", "my_lib", "wife_lib" etc)

[ilovejedd]

Originally Posted by BetterRed

@ilovejedd - does FFS write anything to the source and target device, I'm thinking of its version history, logs etc.

Goodsynch writes its stuff to hidden directories at the root of the source and destination devices - i.e out of harms way, and where they can do least harm.

However I recall seeing another file sync tool that sprinkled stuff in the directories selected for syncing. So if you mirrored D:\Calibre\Calibre Library to E:\Calibre\Calibre Library, it would write 'control' data into those directories which would generate errors in Check Library. I can't recall if FFS was the culprit or the solution.

The sync.ffs_db is optional. It's only created if you select certain settings. I think it was if you enable keep track of moved files? In any case, it's just a single file and it'll be in the same location as metadata.db so quite easy to add to exceptions for Check Library.

Logs are stored in the user's AppData folder by default, iirc.

As mentioned, FFS does create a temporary lock file during sync which gets deleted right after.

[theducks]

Extra files generate warning MESSAGES. In this case, they do no harm, do not indicate 'problem' book entries. I even believe there is an option to 'ignore in the future". As ALWAYS: 'Ignores" should be used with KNOWLEDGE and care

[AnotherCat]

Originally Posted by BetterRed

...However I recall seeing another file sync tool that sprinkled stuff in the directories selected for syncing. So if you mirrored D:\Calibre\Calibre Library to E:\Calibre\Calibre Library, it would write 'control' data into those directories which would generate errors in Check Library. I can't recall if FFS was the culprit or the solution.

The already mentioned file sync.ffs_db gets written into the root of the synchronized library folder by FreeFileSync on both the source and target devices, so is in the same location as metadata.db. Calibre's Library Maintenance shows it as an Invalid Author but if one knows one is using a file sync application it is blindingly obvious what it is. Apart from that Calibre ignores it. If one deletes it the only outcome is that FreeFileSync will just create another next sync session with no issue that I have ever come across (and if there is some sync conflict FreeFileSync will resolve it or ask for guidance).

As already said, one can just tell Calibre Library Maintenance to ignore sync.ffs_db if wanted; I don't bother to do so.

EDIT: in case of confusion (or of a visiting pedant), I should make clear that by "source device" I mean the device that the sync is run from e.g. that which FreeFileSync is running on.

[BetterRed]

Originally Posted by mbovenka

The solution, I think. I use FFS to backup my Calibre setup, and I have no issues with Check Library. It puts a lock file in the source directory, but removes it when done.

Thanks

Originally Posted by ilovejedd

The sync.ffs_db is optional. It's only created if you select certain settings. I think it was if you enable keep track of moved files? In any case, it's just a single file and it'll be in the same location as metadata.db so quite easy to add to exceptions for Check Library.

Logs are stored in the user's AppData folder by default, iirc.

As mentioned, FFS does create a temporary lock file during sync which gets deleted right after.

That was it. Seeing that file name, and after a sleep, I can recall the details.

A fellow member of the local b&m library reading group was getting IO errors on one of his metadata.db files, another member suggested he delete "the database" and run the Library->Maintenance->Restore database.

He deleted metadata.db and sync.ffs_db. After running the calibre database restore everything was fine with calibre. But when he ran FFS it complained about its missing database. That's when he called me.

He's a law professor, with limited IT knowledge. He wanted the archive/versioning features of FFS for his Documents folder (his calibre libraries are at %appdata%\Calibre\Libraries). After I gave him some choices he opted to switch to Goodsync, which doesn't write in user folders and uses NTFS' VSS feature rather than lock files.

Originally Posted by theducks

Extra files generate warning MESSAGES. In this case, they do no harm, do not indicate 'problem' book entries. I even believe there is an option to 'ignore in the future". As ALWAYS: 'Ignores" should be used with KNOWLEDGE and care

Calibre ain't always the only kid on the block. The lawyer thought it better to invest the price of half a bottle of Whisky to keep the backup tool's thumb out of calibre's pie

@AnotherCat - we x-posted

BR

[AnotherCat]

Originally Posted by BetterRed

...He's a law professor, with limited IT knowledge. He wanted the archive/versioning features of FFS for his Documents folder (his calibre libraries are at %appdata%\Calibre\Libraries). After I gave him some choices he opted to switch to Goodsync, which doesn't write in user folders and uses NTFS' VSS feature rather than lock files.

FreeFileSync has the option of VSS also should one have a need for it.

[BetterRed]

Originally Posted by AnotherCat

FreeFileSync has the option of VSS also should one have a need for it.

AAMI - would that obviate the need for 'lock' files. IIRC Goodsync touts its use of VSS as the means by which it can backup files that another program has open for write.

BR

[AnotherCat]

Originally Posted by BetterRed

- would that obviate the need for 'lock' files...

I don't use VSS as I never have any files locked when I sync - so I cannot say from experience. But have just checked the FreeFileSync manual and it confirms it copies locked files in the shadowed volume.

As an aside I don't know the reasoning why FreeFileSync places a file in each synced directory (on both devices being sync'd) and never pondered on it before (the files have never bothered me) but one reason may be that it somehow facilitates its avoiding copy/writes when syncing moved files???

[BetterRed]

Originally Posted by AnotherCat

I don't use VSS as I never have any files locked when I sync - so I cannot say from experience. But have just checked the FreeFileSync manual and it confirms it copies locked files in the shadowed volume.

Likely as not it doesn't need the 'lock' file if its configured to use VSS

Originally Posted by AnotherCat

As an aside I don't know the reasoning why FreeFileSync places a file in each synced directory (on both devices being sync'd) and never pondered on it before (the files have never bothered me) but one reason may be that it somehow facilitates its avoiding copy/writes when syncing moved files???

Goodsynch handles most 'moves' like calibre does: creates a hard link in new location and removes the link in old location. The detection happens in its Analyse phase. Its not a 100%, if you rename a file and move it a million miles it will do a copy and delete. But it will detect, and do a hardlink shuffle, if a book in my Intake library gets a new name and then gets moved to my Books library.

My Books and Intake libraries have a common parent, Calibre\Libraries - that may be the limiting factor, if you move a file from one root level directory to another it may not recognise it as a move, in which case it will do a copy and a delete.

It keeps its state and history data in its root level _gsdata_ hidden folders on source and target devices, I've never had a need to delve into them, in recent times they added a GoodSync Explorer program for doing that.

We're having a hailstorm - marble size so far, damn noisy still.

BR

[ilovejedd]

Originally Posted by BetterRed

Likely as not it doesn't need the 'lock' file if its configured to use VSS

Note, the purpose of the lock file is to let other FreeFileSync instances know that the directory is currently being modified by FFS. That way, you don't have multiple FFS instances trying to make changes to the same folder at the same time as that can mess up the sync (particularly if doing 2-way). A sync.ffs_lock file is created in the main folder which gets deleted immediately after sync is done.

[Rbneader]

Going forward, why don't you just use a portable version of Calibre as your master version? Then all you have to do is copy the Calibre folder. It's what I've done for years.

[ilovejedd]

Originally Posted by Rbneader

Going forward, why don't you just use a portable version of Calibre as your master version? Then all you have to do is copy the Calibre folder. It's what I've done for years.

Lol, this is what I do. Makes it much easier to switch computers.

[ilovejedd]

Originally Posted by AnotherCat

As an aside I don't know the reasoning why FreeFileSync places a file in each synced directory (on both devices being sync'd) and never pondered on it before (the files have never bothered me) but one reason may be that it somehow facilitates its avoiding copy/writes when syncing moved files???

It is precisely for detecting moved files. By default, a "Mirror Sync" doesn't create sync.ffs_db. It's only created when you do a "Two-way Sync" or explicitly enable the "Detect moved files" option when "Versioning".

[Rbneader]

Originally Posted by ilovejedd

Lol, this is what I do. Makes it much easier to switch computers.

It really does!

Actually, all my files - books, movies, documents, everything - are on an external hard drive that I can take with me easily. It makes my life a lot easier, especially as onboard storage has gotten smaller in computers.

It also makes doing backups a lot easier; I just sync the contents of my regular drive to my backup drive once a week or so. Dead simple with any file copy software.