Thoughts on a CC backup/restore function

[chaley]

I am starting to think seriously about a backup/restore function for CC. I have several questions:

First: what should be backed up? There are three things in CC: the settings, the database, and the books. Settings can be subdivided to general settings, folder settings, and calibre device ID settings.

Second: what should be restored? How much granularity is the user given? This is important because some things interact, for example folder settings, the database, and (of course) the books.

Third: what is the most probable use? Catastrophic recovery after wiping a device? Recovery after replacing a device? Cloning settings onto a new device while the originating device is still in use?

Comments are welcome.

[kaufman]

Moderator Notice
Copied from the wish list thread

Wish item for the next release cycle:

Would it be possible to add a function to back up and restore the current settings? You have enough of them now that it would be great to have a way to set up a new device easily. I'm not looking to back up the book database, just the other settings.

Ideally, you would do this my using the google data backup stuff, but I would be almost as happy with the ability to write out and read in a file.

[chaley]

Moderator Notice
Copied from the wish list thread

Originally Posted by kaufman

Wish item for the next release cycle:

Would it be possible to add a function to back up and restore the current settings? You have enough of them now that it would be great to have a way to set up a new device easily. I'm not looking to back up the book database, just the other settings.

This is part of the analysis we are doing for general backup. One of the problems we must solve is migration. What happens if a person restores a backup from 2 or 3 versions ago? We know what to do with the db, run it sequentially through the upgrades. We need to be sure that we do the same for preferences.

Ideally, you would do this by using the google data backup stuff, but I would be almost as happy with the ability to write out and read in a file.

Would Google's backup service really be the right thing to use? It seems to depend on the files being identical across all devices, which is clearly not true for CC. I can't find any way to control what data set get restored or to have multiple data sets.

There is also a lot of chatter on the 'net saying that the service is fundamentally unreliable. I don't know the truth of that, but given that backups and restores "magically happen" at unknown points, I can see lots of room for problems.

[kaufman]

Moderator Notice
Copied from the wish list thread

Originally Posted by chaley

This is part of the analysis we are doing for general backup. One of the problems we must solve is migration. What happens if a person restores a backup from 2 or 3 versions ago?

To me, this makes the most sense for backing up settings. People can always transfer their books again. If it was an older version of the backup, you could just apply the settings that still exist and ignore the other ones.

I wasn't thinking of the backup as strict copy of the internal data structures, but as a set of preference pairs that you could reapply as the file was loaded. I used an app once that generated this in text as an email that could be copy/pasted into the app later. Then the app parsed the text and reset the settings.

Having said all this, if you wanted to backup the book database too, I wouldn't mind. Although, then you would also have to backup the books for people who keep them on internal storage.

It seems to me that there are two use-cases for this.

1) I lost my device
2) I am upgrading my device.

If you are only doing this to support usecase 2, you can just insist that the user upgraded to the current version before starting the migration. For people with books on an external SD card the work would be:

1) Make sure CC is up to date on old device.
2) backup settings and book db to SD card
3) Move SD card to new device
4) Install CC on new device
5) Load settings and book database

For people without an SD card, I am not sure what the easiest way to do 2 and 5 would be. Plus you would also have to backup and restore the books for them.

For usecase 1, the settings book db and books would need to be stored off device to be of any use, and since (except for the settings) they are already in calibre, would anyone really do that?

I think I am back to my first position. Just backup and restore settings. Its small enough to move around, and gives most of the value that people might be looking for.

[chaley]

Originally Posted by kaufman

To me, this makes the most sense for backing up settings. People can always transfer their books again. If it was an older version of the backup, you could just apply the settings that still exist and ignore the other ones.

The one counter argument I can make is that the user might not remember what books were on the device. If the database were restored then the books would be listed in calibre on the next connect. The user could select them all and send them. If I ever do the "auto-send changed book" feature then this would become automatic. Connect and the books arrive.

I wasn't thinking of the backup as strict copy of the internal data structures, but as a set of preference pairs that you could reapply as the file was loaded. I used an app once that generated this in text as an email that could be copy/pasted into the app later. Then the app parsed the text and reset the settings.

I would rather not have a pick & choose scheme. Lots of settings are related to each other in non-obvious ways.

Attaching the backup to an email message is an interesting idea. I was assuming that the user would put the file into dropbox or drive or what-have-you. CC would create the file wherever the user wants, then the user would use the cloud storage app of choice to get the backup off the phone.

Having said all this, if you wanted to backup the book database too, I wouldn't mind. Although, then you would also have to backup the books for people who keep them on internal storage.

I am not totally convinced for the reasons mentioned above.

It seems to me that there are two use-cases for this.

1) I lost my device
2) I am upgrading my device.

There are at least two more:
3) I am adding another device while keeping the original (cloning my device).
4) I needed to uninstall CC for some reason and want to restore it.

If you are only doing this to support usecase 2, you can just insist that the user upgraded to the current version before starting the migration. For people with books on an external SD card the work would be:

1) Make sure CC is up to date on old device.
2) backup settings and book db to SD card
3) Move SD card to new device
4) Install CC on new device
5) Load settings and book database

This would also work in my additional use case 4, with the assumption that the books are still in main memory.

Hmmm... I am not sure what happens if I install an SD card on a new machine when an app is using the private SD card app folder. Does the app adopt what is there? Does Android clear it first? My guess is that the folder is adopted or things would go bad very fast every time I swapped cards, but given that the kitcat changes make very little sense (to me) I don't trust my intuition.

For people without an SD card, I am not sure what the easiest way to do 2 and 5 would be. Plus you would also have to backup and restore the books for them.

Send the backup to the cloud. As for books, again same as above. Or send the books to the cloud as well.

One scheme would be to store the settings and DB backup in the Calibre_Companion folder and use Dropsync (or something like it) to upload that folder to Dropbox whenever a backup is made. That makes a self-consistent set of data.

For usecase 1, the settings book db and books would need to be stored off device to be of any use, and since (except for the settings) they are already in calibre, would anyone really do that?

This is a good question. I have already discussed one reason why having the CC database might be a good thing even when I don't have the books. However, it is a very good question whether or not people will understand that backing up the database does not back up the books. Saying that *only* settings are backed up eliminates that possible misunderstanding.

I think I am back to my first position. Just backup and restore settings. Its small enough to move around, and gives most of the value that people might be looking for.

Thanks for the comments.

A question related to backing up the db: should the calibre device ID also be backed up? If you are cloning a device or restoring after uninstall/wipe then restoring the device ID would prevent uploading all the metadata on the first connect after restore. However, if you are copying a device then restoring the device ID would be a bad idea because calibre would not be able to distinguish between them. It is very likely that metadata caching would break.

[kaufman]

Originally Posted by chaley

The one counter argument I can make is that the user might not remember what books were on the device. If the database were restored then the books would be listed in calibre on the next connect. The user could select them all and send them.

Good point. I was thinking that the database would be useless without the books for restore purposes, but I forgot that CC tells Calibre which books are "present" on the device. People would have to know thay they needed to resend the books the Calibre said were already there, but the backup/restore function is probably nerdy enough that people who use it will understand the extra work involved.

Originally Posted by chaley

I would rather not have a pick & choose scheme. Lots of settings are related to each other in non-obvious ways.

I wasn't thinking that the user would pick and choose, but that the program would read the key and use that to set the value internally. As long as a person didn't try to restore a newer version of the backups on an older version of the program, there should never be a setting that didn't make sense.

Originally Posted by chaley

Attaching the backup to an email message is an interesting idea. I was assuming that the user would put the file into dropbox or drive or what-have-you. CC would create the file wherever the user wants, then the user would use the cloud storage app of choice to get the backup off the phone.

Definitely the way to go if backing up the DB. The Email really only works for a small file like the settings.

Originally Posted by chaley

Hmmm... I am not sure what happens if I install an SD card on a new machine when an app is using the private SD card app folder. Does the app adopt what is there? Does Android clear it first? My guess is that the folder is adopted or things would go bad very fast every time I swapped cards, but given that the kitcat changes make very little sense (to me) I don't trust my intuition.

I don't know either, but I would be glad to test this for you when I get my new phone. Or, perhaps I can test it by moving my SD card from my phone to my tablet. If the books files can still be accessed, we should be ok. I'll try that later.

Originally Posted by chaley

One scheme would be to store the settings and DB backup in the Calibre_Companion folder and use Dropsync (or something like it) to upload that folder to Dropbox whenever a backup is made. That makes a self-consistent set of data.

True, but a common use of Dropsync is to do the backup automatically every time things change. Something like this might make for a lot of extra backups. Or a backup while the program was running.

Originally Posted by chaley

A question related to backing up the db: should the calibre device ID also be backed up? If you are cloning a device or restoring after uninstall/wipe then restoring the device ID would prevent uploading all the metadata on the first connect after restore. However, if you are copying a device then restoring the device ID would be a bad idea because calibre would not be able to distinguish between them. It is very likely that metadata caching would break.

I would say no. This whole operation refers to something rare enough that an occasional extra metadata load is probably better then causing some problems in some situations.

[jackie_w]

Just personal opinions ... I would only make use of a Settings backup (not books/database) and I would only use it to set up a new (or factory reset) device. I've already got slightly different settings for my phone and tablet but cloning backed up settings from the most similar device would do most of the work.

Having said that, when I last set up a new device it wasn't very onerous to have the old and new side by side and work through the settings one by one. Setting up the format folders took the longest but a backup wouldn't have helped, because the new device was KitKat and the old one wasn't, so I would have had to set them manually anyway.

[kaufman]

Originally Posted by jackie_w

Just personal opinions ... I would only make use of a Settings backup (not books/database) and I would only use it to set up a new (or factory reset) device. I've already got slightly different settings for my phone and tablet but cloning backed up settings from the most similar device would do most of the work.

I agree with this. Others might have complicated subsets of their books, but I just keep everything on my phone and tablet, so all I need is the settings.

Originally Posted by jackie_w

Having said that, when I last set up a new device it wasn't very onerous to have the old and new side by side and work through the settings one by one.

This was probably more true before Chaley spent the last 6 months adding every crazy option his users could think of. It's certainly still doable, but - if it isn't too much effort to implement - would be nice to have.

[HarryT]

"Settings" backup sounds by far the most useful to me. As the number of settings in CC expands, it becomes increasingly useful. Backing up the books and the DB seems a lot less useful.

[jackie_w]

Originally Posted by kaufman

... but - if it isn't too much effort to implement - would be nice to have.

Agreed

[Dhevi]

I would agree with Kaufman, HarryT and jackie_w. Creating a backup of my app settings would be the most I would use. I don't see a need, for myself anyways, to backup my book files.

An application I use MoneyWise Pro actually provides an option to Create/Restore a Backup of settings/data. I can select to create a local backup which saves to the device internal storage a backup to Dropbox (perhaps an option to allow a user to select a Cloud Storage app). Similarly I can restore a backup from either location. I can select how many backups to keep stored in each location (though I'm not certain that keeping multiple backups might be needed in the case of CC per device).

[chaley]

Thanks to all for the comments. Backing up settings is certainly easier to explain. Backing up the db opens the door to all sorts of misunderstandings, none of which we need.