How to Duplicate My Calibre Library on Multiple Computers?

[Sydney's Mom]

I will always keep an SD and mini hard-drive backup of my library. Octuple redundancy.

[Starson17]

Originally Posted by pilotbob

Nothing you said is wrong.

But can be easily mitigated by being careful (manual). Also, Dropbox in addition to syncing also retains older versions of files. So, if the metadata.db file does get corrupted you can always go back to a previous version.

You could probably also easily mitigate it buy using a .bat file to start calibre. It can write a "inuse.txt" dummy file in the dropbox/calibre... and delete it on exit. In this case if the batch files sees the text file there it will exit with an error that calibre is in use on another PC.

BOb

I agree with all you've written. I have some dummy flag files checked by batch files to mitigate the issue, as you've suggested for similar situations. I've also got a step in my backup that copies the metadata.db to metadata.db.1 etc. up to the last 6.

I wasn't really trying to say don't use dropbox - it looks like a great option, it just needs to be used with some care and knowledge. There's always risk - your hard drive may blowup, too, so the advantages of dropbox can easily outweigh the relatively minor risk. I'd be more tempted if my my library would fit in the free space.

[Starson17]

Originally Posted by daffy4u

I do not fully trust "the cloud" which is why I copy my Calibre library from Dropbox to all my computers (except the desktop which was used to created the Dropbox library). Having a local copy on each machine also gives me access to my books when I do not have an internet connection (as well as being additional back-ups).

Bouncing between the installations I have of Calibre on each computer hasn't caused me a problem so far. Calibre is not open on all computers at the same time.

I'm not sure if you mean you keep a non-dropbox monitored directory on each machine, or just let dropbox sync a copy on each machine.

I probably shouldn't even be discussing this, as I'm sort of guessing how dropbox works, but usually, the cloud holds a copy, and you point that copy to a directory on each machine. If a file gets updated in the monitored directory, the cloud sees that, it pulls a copy into the cloud and sends the updated file to all computers. That means that if you get a corrupted metadata.db file on any one computer, that corruption gets replicated to all.

There are two issues. The first is preventing corruption. Avoiding simultaneous use of Calibre on two computers goes partway to doing that, but as I posted, it isn't enough. A batch file launch with a dummy flag file can also help. The other issue is having a backup to recover from when/if a corruption gets replicated to all monitored directories. Don't expect a local copy to be a backup if that local copy is being monitored and updated by dropbox.

[daffy4u]

Originally Posted by Starson17

I'm not sure if you mean you keep a non-dropbox monitored directory on each machine, or just let dropbox sync a copy on each machine.

I probably shouldn't even be discussing this, as I'm sort of guessing how dropbox works, but usually, the cloud holds a copy, and you point that copy to a directory on each machine. If a file gets updated in the monitored directory, the cloud sees that, it pulls a copy into the cooud and sends the updated file to all computers. That means that if you get a corrupted metadata.db file on any one computer, that corruption gets replicated to all.

There are two issues. The first is preventing corruption. Avoiding simultaneous use of Calibre on two computers goes partway to doing that, but as I posted, it isn't enough. A batch file launch with a dummy flag file can also help. The other issue is having a backup to recover from when/if a corruption gets replicated to all monitored directories. Don't expect a local copy to be a backup if that local copy is being monitored and updated by dropbox.

I'm sure there may be instances where one would have multiple copies of Calibre open at the same time but I can't think of a situation where I have or need to do that. I'm only working on one computer at a time, so I haven't had an issue.

Until BOb enlightened me I did have additional full copies of my library on all machines (Desktop = original + Dropbox, Laptop = Copy + Dropbox, Netbook = Copy + Dropbox). Now I've dropped the copies on my Laptop and Netbook (Desktop = original + Dropbox, Laptop = Dropbox, Netbook = Dropbox). My Desktop is backed-up constantly by Carbonite as well (and my Kindle 1 still holds all my Kindle books plus an FTP copy on my personal website that get updated occasionally).

[Starson17]

Originally Posted by daffy4u

I'm sure there may be instances where one would have multiple copies of Calibre open at the same time but I can't think of a situation where I have or need to do that. I'm only working on one computer at a time, so I haven't had an issue.

I was assuming you never did that. That would almost certainly corrupt your database. My worry is about other issues. Perhaps you worked on computer1, then shut down. Now you open your laptop and begin working on that, but it hasn't made an internet connection. Then it connects. It will not have changes from computer1, but will have changes from the laptop. You will end up with either corruption or lost data. As I said, it's only in unusual circumstances where you have a problem, so not having an issue is the norm ... until you do.

Until BOb enlightened me I did have additional full copies of my library on all machines (Desktop = original + Dropbox, Laptop = Copy + Dropbox, Netbook = Copy + Dropbox). Now I've dropped the copies on my Laptop and Netbook (Desktop = original + Dropbox, Laptop = Dropbox, Netbook = Dropbox). My Desktop is backed-up constantly by Carbonite as well (and my Kindle 1 still holds all my Kindle books plus an FTP copy on my personal website that get updated occasionally).

Which directory did you point your Calibre to? If you've pointed to the dropbox directory, and deleted the original, you've removed one layer of backup protection, but added convenience. As long as your other backups are good, you should be fine and the convenience may be worth it.

[pilotbob]

Originally Posted by Starson17

My worry is about other issues. Perhaps you worked on computer1, then shut down. Now you open your laptop and begin working on that, but it hasn't made an internet connection. Then it connects. It will not have changes from computer1, but will have changes from the laptop. You will end up with either corruption or lost data.

I think you aren't giving Dropbox enough credit. It is extremely robust. Are you able to duplicate this situation?

BOb

[Starson17]

Originally Posted by pilotbob

I'm not sure what it does if there is an existing library there. But, I would assume it would just use that one from now on.

Correct - it just uses the new library. If there's a metadata.db file there, it just uses that. I keep a small library for testing changes to the Calibre code and my normal one and switch between them several times a day. (My wife gets annoyed if I don't leave the main one open when she wants access to her news and books.

[daffy4u]

Originally Posted by Starson17

Which directory did you point your Calibre to? If you've pointed to the dropbox directory, and deleted the original, you've removed one layer of backup protection, but added convenience. As long as your other backups are good, you should be fine and the convenience may be worth it.

To quote myself:

Until BOb enlightened me I did have additional full copies of my library on all machines (Desktop = original + Dropbox, Laptop = Copy + Dropbox, Netbook = Copy + Dropbox). Now I've dropped the copies on my Laptop and Netbook (Desktop = original + Dropbox, Laptop = Dropbox, Netbook = Dropbox). My Desktop is backed-up constantly by Carbonite as well (and my Kindle 1 still holds all my Kindle books plus an FTP copy on my personal website that get updated occasionally).

Outside of Dropbox I have multiple other copies of my library.

[Starson17]

Originally Posted by pilotbob

I think you aren't giving Dropbox enough credit. It is extremely robust. Are you able to duplicate this situation?

BOb

In what way is it robust? This is just how sync programs work. They can't integrate changes from two machines into a database unless they know exactly how that database works and I'm certain dropbox does not know how Calibre works. All they can do is copy the database files.

Cloud based sync programs can't update changes to or from the laptop if the laptop hasn't got a connection to the cloud. If you work on the laptop when you don't have an internet connection (as someone said they wanted to do so they didn't need to rely on the built in content server), those changes don't appear on any other dropbox machine. They sit on the laptop until it gets connected to the cloud.

If the owner comes home and works on Calibre there, you've got a conflict at the least. If you're working on Calibre at home and someone turns on the laptop, the home version may have changes in memory, just as dropbox tries to update the home copy of metadata.db as the laptop comes online. I suspect that will lead to corruption for the same reason that running two version of Calibre at the same time will.

Someone here (you?) said it posted a "conflicted" copy in the first situation. That's what I would expect, and it's a help. You could rename the conflicted copy of metadata.db, export the opf file for missing records, switch back to the other metadata.db and import to recover the lost data. A corruption would be more of a problem, and that could end up being replicated to everything. As I said, I'm not trying to denigrate dropbox. It just needs to be used with some care and knowledge.

Do you have any reason to believe it doesn't work as I described?

[Starson17]

Originally Posted by daffy4u

To quote myself:

Outside of Dropbox I have multiple other copies of my library.

Right. I read that, and it allows you to restore to the time of the last backup to those copies.

Again, I'm not saying don't do this. It's really convenient. Just be aware that the risk is slightly increased. You have to judge if that risk is worth the convenience. Use it wisely.

Know that if you've made changes on your laptop or home computer, they need to have a connection to the web, and dropbox needs to be responding and have time to receive those changes and send them to the next computer in the chain before you start working on that next computer.

Be cautious about firewalls blocking access to dropbox. Think about daylight saving time changes that may change the dates on two different computers.

Don't rely solely on a backup of your computer made yesterday if that's just a backup of the dropbox account. Make a separate backup of a good and complete copy of the Calibre library, or at least the metadata.db file, which is the most important file in Calibre.

[chaley]

Originally Posted by pilotbob

I think you aren't giving Dropbox enough credit. It is extremely robust. Are you able to duplicate this situation?

I am part of a multi-country multi-partner project at my university, and we use dropbox to share code and papers under construction. We have frequently (several times per week) had cases where changes were simultaneously made to a file in different dropboxes. The result was the creation of 'conflict' files, all of which needed to be manually examined to determine which one was correct. In a couple of cases, neither were correct; we had to merge them together.

The worst case we encountered happened in an eclipse java project. A person in Italy and a person in the UK each changed a different fundamental class at (more-or-less) the same time. The change caused the regeneration of a large number of .class files . The result: because of dropbox propagation delays some files came from Italy and some came from the UK. Nothing worked. We had to get on the phone, stop one of the people, regenerate, then let dropbox propagate. Then do the same thing in Italy. Once everything was back in sync, we could continue to work on our individual parts.

[pilotbob]

Originally Posted by Starson17

In what way is it robust? This is just how sync programs work. They can't integrate changes from two machines into a database unless they know exactly how that database works and I'm certain dropbox does not know how Calibre works. All they can do is copy the database files.

Well, actually Dropbox doesn't work at the file level... it creates a patch of the binary changes.

I mean, it is robust in that if it sees that the file it has a change for doesn't match the same base that it got the change from it won't apply the change. So... the corruption you are concerned about won't happen. Sure... they will get out of sync and you will have to remedy it.

Originally Posted by Starson17

Do you have any reason to believe it doesn't work as I described?

No, I am saying it won't cause corruption which is what you stated. Also, you won't have any "data loss" either... the data will still exist where you made the original change. You just have to manually reconcile the conflict.

https://www.dropbox.com/help/36

BOb

[Starson17]

Originally Posted by chaley

Once everything was back in sync, we could continue to work on our individual parts.

And I'll bet you continue to use dropbox. The convenience is great, but you have the skill and the knowledge to recover from the relatively infrequent problems, so it's more than worth it for you.

I feel like the guy throwing cold water on everyone, when all I want to do is make sure that anyone who's doing this has some idea of how it works so they can avoid problems.

[Starson17]

Originally Posted by pilotbob

Well, actually Dropbox doesn't work at the file level... it creates a patch of the binary changes.

I mean, it is robust in that if it sees that the file it has a change for doesn't match the same base that it got the change from it won't apply the change. So... the corruption you are concerned about won't happen.

I'm worried about a change in memory that Calibre is holding as you work, but that Calibre has not yet written to the file. I'm worried that dropbox will change the file after Calibre has read it, but before it has written the change you just typed on the screen and which is only in memory. I worry that this possibility is why Kovid says don't run two copies of Calibre at the same time.

Maybe I worry too much and corruption isn't an issue. Maybe Calibre locks all files early enough that dropbox would never step on any of the parallel threads Calibre has running. I just don't know. Are you sure you do?

Sure... they will get out of sync and you will have to remedy it.

And this is something that less sophisticated users need to be aware of. Then they can choose if they want this risk and have the ability to deal with it. Maybe they don't care about occasional bits of data loss that need to be manually recovered by reconciling conflicts and they can ignore it.

[daffy4u]

Originally Posted by Starson17

Don't rely solely on a backup of your computer made yesterday if that's just a backup of the dropbox account. Make a separate backup of a good and complete copy of the Calibre library, or at least the metadata.db file, which is the most important file in Calibre.

I think you're assuming that I live in Calibre. It is not something I use daily. If I use it more than a couple of times month, that's a lot (though I do go through conversion frenzies from time to time).

This year I'm trying to read more of the freebies I've collected over the past couple of years, most have already been converted or don't need conversion as they are Kindle books (I keep DRM stripped versions in Calibre but I read the DRM'd versions on my Kindles). I can still drag and drop those books onto my computer from Windows Explorer without opening Calibre.