How to Duplicate My Calibre Library on Multiple Computers?

[kovidgoyal]

Originally Posted by chaley

I am part of a multi-country multi-partner project at my university, and we use dropbox to share code and papers under construction. We have frequently (several times per week) had cases where changes were simultaneously made to a file in different dropboxes. The result was the creation of 'conflict' files, all of which needed to be manually examined to determine which one was correct. In a couple of cases, neither were correct; we had to merge them together.

Interesting (and slightly horrifying) use for dropbox, why don't you use a version control system?

On the question of robustness. I'm not a dropbox user and I really don't know too much about it, but as Starson17 pointed out, calibre relies on an sqlite database. I would be rather surprised if dropbox was able to handle concurrent changes to the database file, which to it is presumably an opaque binary blob.

In addition, consider the following scenario of two users making concurrent changes to the same calibre library.

User 1 changes the author name of an entry. This updates the db for user 2. calibre does not rescan the db everytime the file changes, so User 2's calibre instance will continue to happily use its cached data. Now suppose user2 also decides to edit the same author name. calibre2 will use the old author name before user1's edit and will therefore not be able to find the files fo that book, since the file location depends on the author name. It will therefore think that entry has no files and create a new folder for that book with no files in it.

Now when dropbox syncs, both users will have lost the files of that entry, which will be in an orphaned folder based on the author name user 1 chose.

I can think of lots of problems like this caused by concurrent edits. Really the only safe way to support concurrent editing is with row level locking, the way it is done in databases.

So please dont use multiple calibre instances on the same dropbox share concurrently, or I am going to be dealing with a lot of reports about missing files

[chaley]

Originally Posted by kovidgoyal

Interesting (and slightly horrifying) use for dropbox, why don't you use a version control system?

Short, common, story.

The project needed to produce a 'deliverable document' in a couple of weeks. Each partner was in charge of a section. One of the project members set up a partitioned document structure using latex sub-documents, put this into dropbox to simplify sharing the parts, and announced its existence to the partners. It worked quite well. So far, so good.

As days went on, the document structure started to morph. People found that 'their part' was actually distributed through the document. Clearly there would be simultaneous editing, but "it wouldn't be too much and anyway, who has time to set up a repository? We must finish soon!" This didn't work as well, but we could handle the conflicts.

Then someone decided that some code was needed to generate some of the figures, and as a side effect do some formal analysis and proofs on parts of the model. Great idea. Bang into dropbox it went along with a logic engine and a bunch of other stuff, and bang went the collisions. However, we were 2 days from deadline, so we lived with it.

This story is a classic example of why 'we' should think about what we are doing before charging off to use some shiny new doodad that seems to solve the problem. Although dropbox is trivially easy to use, distributed repositories are also easy to use and are more appropriate for the task. A subversion repository coupled with tortoiseSVN would have neatly solved the problem, and is arguably just as easy to use especially in this context.

Sigh.

[Starson17]

Originally Posted by daffy4u

I think you're assuming that I live in Calibre. It is not something I use daily.

I'm not really assuming anything about you. I just hoped people would use care with setting up dropbox, regardless of how often they use Calibre.
For those who use it very seldom, the risk of concurrent use and conflicts seems to me to be very, very, low. That's one of those factors to be considered when weighing convenience against risk.

[Sydney's Mom]

Originally Posted by kovidgoyal

So please dont use multiple calibre instances on the same dropbox share concurrently, or I am going to be dealing with a lot of reports about missing files

Well, I had missing files tonight, but I knew it was dropbox, and not Calibre, so I didn't bother you, Kovid.

I did just what everyone said not to do. I had uplifted everything to the clouds, and about 80% was downloaded, and I just couldn't resist peeking at my netbook. My laptop was shut down, as was my desktop. Apparently, the metadata file hadn't downloaded yet, and so my library showed no books. Oh well, I thought, when it is 100% done, they will be there. Well, when I got home from work, dropbox was done syncing, but my laptop still showed XXX books, and my netbook showed 0. I was about to uninstall Calibre (the only thing I could think of was to reinstall), but chaley warned me about creating a metadata file by opening calibre on my netbook. Sure enough, when I opened the dropbox file, there was a metadata file with 75 kb, and a metadata conflicted file with 4000 kb. I deleted the smaller, and renamed the larger, and I was in business! Then to the desktop. Same issue (but all other Calibre libraries were closed). I opened the dropbox file on the desktop, and saw the 75 kb metadata file, but no larger file. I went to my file in the clouds, and downloaded and saved the larger file to my desktop. It worked! So Kovid, I hope I can keep using dropbox. I promise, if I lose my books again, I won't blame Calibre. I have a noncloud copy on an SD card and a portable hard drive. I should be okay. But I like to go into Calibre from a couple different computers. It is great not to have to email the book back and forth. And I finally feel I have enough backups (SD, HD, Dropbox, laptop, netbook, desktop). Sectuple rendundncy?

[ficbot]

I am only going to use it on my Macbook. If I want to access my documents from work (it is unlikely I would need to access my ebooks there) I will log in through their website.

[DoctorOhh]

Originally Posted by Starson17

Make a separate backup of a good and complete copy of the Calibre library, or at least the metadata.db file, which is the most important file in Calibre.

I use Dropbox for syncing and I think you have been correct in your assumptions of possible pitfalls. I've used it now over a year and I have found most of the problems have been inflicted by me.

I have two backup strategies.

1. Log into the kids computer under my account and let Dropbox sync the database. When it is finished I log off. I never use Calibre on this machine and Dropbox does not automatically start up. Setting this up this way protects me from my own errors which are far worse then any hypothetical Dropbox conflict you can come up with.

2. The backup strategy I use outside of Dropbox doesn't involve backing up the metadata.db file at all.

Update: The below method will not backup custom recipes, they are stored in the metadata.db file. I save all of my custom recipes separate from Calibre but I wanted folks to know the limitation of this backup option.

• I use the Save to Disk option in Calibre (ensuring both Save cover separately and Save metadata in OPF file are checked) to save my books in a backup directory.
• Every time I add/update any books I select them and use Save to Disk to save them to my backup directory.
• If I ever need to restore from a catastrophic failure I just reinstall Calibre and point it to a new Library folder and import the books. All the metadata, covers, tags, series etc... are recreated perfectly.

I have actually restored using 2 above a couple of times to test this setup and my library has always past the disaster recovery test.

Everyone here needs to know that a backup isn't a backup unless it can be restored. Far too many times folks thought they had a backup strategy only to find out is wasn't as good as they thought when they actually had to restore data.

Confession, like most other paranoids in this thread I do drag my library to an external drive about once a week for good measure and all of my source files are far away from Cailbre and in multiple locations in case of operator hardware error.

[kovidgoyal]

@sydney's Mom: Feel free to continue using drop box, but just be aware of the pitfalls

[SirBC]

Originally Posted by kovidgoyal

I can think of lots of problems like this caused by concurrent edits. Really the only safe way to support concurrent editing is with row level locking, the way it is done in databases.

Is it possible to have Calibre run in a "read only" mode, so that it didn't attempt to write to the DB? I guess I'm thinking of a scenario where, say, one family member is the one who maintains updating the ebooks in Calibre while the other family members just browse.

[kovidgoyal]

That's what the content server is for.

[SirBC]

I haven't yet looked into the content server but I will. Do the people viewing via the server have the same full functionality as within Calibre? I keep my Calibre DB on my NAS, so other family members can access even when my computer is off. Would that still be true with the content server?

[Starson17]

Originally Posted by SirBC

Do the people viewing via the server have the same full functionality as within Calibre?

No. The content server provides access to the books via the web. You can search and download, but not change metadata or add/delete books.

[chaley]

Originally Posted by SirBC

I keep my Calibre DB on my NAS, so other family members can access even when my computer is off. Would that still be true with the content server?

Note that a computer is required to run the content server. Unless your NAS server can run Calibre (perhaps it is a linux box?), you would need to leave the calibre server running on a NAS client.

[Starson17]

Originally Posted by SirBC

I keep my Calibre DB on my NAS, so other family members can access even when my computer is off.

How do you prevent two people from using it at the same time?

[SirBC]

Originally Posted by Starson17

How do you prevent two people from using it at the same time?

Currently I don't. That is why I was asking if it was possible to have Calibre start in a "read only" mode so that my other family members could use Calibre to access the DB when my computer is off, as well as do things like use tags to filter books. It sounds like the content server would kind of work, but we would have issues with limited functionality (if I am correctly understanding how content server works) as well as not being able to access the content server when my computer is off. It's not that a big a deal, I was just wondering if there was a startup command to run Calibre in read only mode to avoid possibly corrupting the database.

[Starson17]

Originally Posted by SirBC

Currently I don't. That is why I was asking if it was possible to have Calibre start in a "read only" mode so that my other family members could use Calibre to access the DB when my computer is off,

Currently it's not possible to start in a read only mode.

 as well as do things like use tags to filter books.

You can do this through the content server.

It sounds like the content server would kind of work, but we would have issues with limited functionality (if I am correctly understanding how content server works)

What functions do you want? The content server can get all books/news and you can search by tags, author, title, etc. to find the content.

as well as not being able to access the content server when my computer is off.

This is correct. OTOH, you gain the advantage of access anywhere in the world where there is internet access and access to your books via smartphone and other devices.

It's not that a big a deal, I was just wondering if there was a startup command to run Calibre in read only mode to avoid possibly corrupting the database.

You could write a batch file that checks for the existence of a flag file, exits if it exists, creates it if it does not, then deletes it when you exit Calibre. If everyone runs it to start Calibre, they'd avoid stepping on each other.