File lock for multiuser Calibre ???

[rjalex]

We are a small group of friends that want to mantain a common Calibre library on cloud storage (DropBox).

Calibre would be installed on each of our laptops pointing to the local Dropbox folder.

Each of us can import a book to our library using his Calibre instance.

Each of us can correct any book metadata, add cover images, add tags.

Do you have suggestions on how to handle file locking so that ?

Thank you very much.

[theducks]

Originally Posted by rjalex

We are a small group of friends that want to mantain a common Calibre library on cloud storage (DropBox).

Calibre would be installed on each of our laptops pointing to the local Dropbox folder.

Each of us can import a book to our library using his Calibre instance.

Each of us can correct any book metadata, add cover images, add tags.

Do you have suggestions on how to handle file locking so that ?

Thank you very much.

Rewrite Calibre to be Multi-User while leaving it available for (novice) single user systems.

Calibre is Single User at a time


The 'Content Server' allows multiple read access.

If you want Write access, you might consider making a dedicated 'Calibre PC' available for Remote Desktop sharing (with gobs of access security) that allows a single-at-a-time logon. (This PC needs some CPU/Memory 'chops' as it will be doing all the work.)

[kiwidude]

If you genuinely need simultaneous write then as theducks has said your options are limited.

However what you "could" do is just treat dropbox as an add source/save to disk target. So you save to disk out of Calibre onto the DropBox location (making sure you save the opf/jpg). Then others can use Add from Folder, with automerge turned on. So that way you will pickup any metadata changes/new covers/versions etc others put in.

The issue with that is if the author/title is incorrect and gets changed so you get duplicates. So what I would do is have two folder areas - one for the "unprocessed" books and one for the "clean" ones. So you only ever import into your local library from the "clean" area to prevent duplicate issues from renaming.

Then the workflow becomes someone picks a folder of books from the "unprocessed" area and removes it from there to their local disk (so others dont try to clean them at the same time). Then import from that folder to Calibre, clean them up and then save to disk back into the "clean" folder in Dropbox for others to import, either just that author if you are communicating with each other or the whole clean folder if you use the automerge approach.

Sounds a nice way to share the burden of getting a clean library put together.

[DoctorOhh]

To the OP --- tread carefully.

Originally Posted by kiwidude

However what you "could" do is just treat dropbox as an add source/save to disk target. So you save to disk out of Calibre onto the DropBox location (making sure you save the opf/jpg). Then others can use Add from Folder, with automerge turned on. So that way you will pickup any metadata changes/new covers/versions etc others put in.
....
Then the workflow becomes someone picks a folder of books from the "unprocessed" area and removes it from there to their local disk (so others dont try to clean them at the same time). Then import from that folder to Calibre, clean them up and then save to disk back into the "clean" folder in Dropbox for others to import, either just that author if you are communicating with each other or the whole clean folder if you use the automerge approach.

The dropbox is a local folder and each of the three machines. they can each be working on the library at the same time. Even if they were working on different books when each of their metadata.db files tries to update the main cloud (and in turn each of their local machines) it is very probable that they will experience blocked files.

Using dropbox in this manner is only a viable option if they work out some sort of time schedule. If they ever update close to the same time they will be frustrated with missing books that get locked as conflicted copies on their local machines.

[kiwidude]

@dwanthny - you misunderstand me I think. They would have their own Calibre library that is not in dropbox - the metadata.db is not shared. As I said above they just use Dropbox to Add from as a source folder into their local library. No concurrency issues whatsoever.

[DoctorOhh]

Originally Posted by kiwidude

@dwanthny - you misunderstand me I think. They would have their own Calibre library that is not in dropbox - the metadata.db is not shared. As I said above they just use Dropbox to Add from as a source folder into their local library. No concurrency issues whatsoever.

I understood you. Do you use Dropbox? If they all decided to copy to their dropbox (local machine) at or close to the same time then it is probable that they will cause the metadata.db file in the dropbox to get conflicted and not update the cloud or another's local copy of the dropbox metadata.db file.

Update: its not a calibre concurrency issue it is a dropbox concurrency issue.

[kiwidude]

Yes I do use dropbox. There is no metadata.db file stored there with what I propose.

They have a local Calibre library on their machine, not within a dropbox folder. The dropbox folders are populated using save to disk exporting from their library - so dropbox will contain opf/jpg and book files, but no metadata.db.

[DoctorOhh]

Originally Posted by kiwidude

Yes I do use dropbox. There is no metadata.db file stored there with what I propose.

They have a local Calibre library on their machine, not within a dropbox folder. The dropbox folders are populated using save to disk exporting from their library - so dropbox will contain opf/jpg and book files, but no metadata.db.

Sounds like a plan.

I guess its time for me to go to bed.

[kovidgoyal]

I would suggest the following trick. Write a small batch file/script to launch calibre. Have the batch file check for the existence of a dummy file in the dropbox folder if it exists it should abort, if it doesn't it should create it and launch calibre. Once calibre quits it should delete the file.

Note that this isn't fully robust since the file isn't a proper lock file, but it might be "good enough" if only a small number of people are doing this.

Backup your calibre library often.

[rjalex]

Thank you all.

Yes I had thought about the "manual lock file" idea. We are three friends and we would impose the discipline to write a "namemmdd.lock" file (e.g. "bob0422.lock") on dropbox and delete it when we finish using it, and the batch file could be a great idea to add. The purpose of writing name/date is that if one of us "forgets" the lock there we can ring him/her up ask as to why and then carry on

Also the "dropbox as data vault out of calibre's jurisdiction with a "clean" and "to fix" directories" is a clever idea. If I get your idea well the "to fix" data would be "added" to a non-dropbox local librabry adn when clean exported "to disk" where the disk is the "clean" book with the mobi, opf etc files ? The only part I am not quite understanding is then how to use cailbre to load the clean books on my kindle (very new user hence I might be unaware of Calibre being able to use two different libraries).

Thanks to all and happy Easter.
Bob

[kiwidude]

Originally Posted by rjalex

Also the "dropbox as data vault out of calibre's jurisdiction with a "clean" and "to fix" directories" is a clever idea. If I get your idea well the "to fix" data would be "added" to a non-dropbox local librabry adn when clean exported "to disk" where the disk is the "clean" book with the mobi, opf etc files ? The only part I am not quite understanding is then how to use cailbre to load the clean books on my kindle (very new user hence I might be unaware of Calibre being able to use two different libraries).

Whether you have two local libraries or one is up to you and whether you want to mix the batch of books you are working on cleaning up currently with the books that you have cleaned so far. In terms of getting books onto your kindle it would be as per usual of doing send to device from your local library. You would likely not want any unclean books on there. So either you could use a tag or custom column when you add books from the "to fix" folder to your single local library which you remove as you clean them up. Or you could have two local libraries, one for your to fix books and one for your clean ones. It is trivial with calibre to switch libraries and to move books between them.

So the choice is up to you and each of you friends can make their own decision as the libraries are local and not shared. The important thing is just the save to disk back into the "clean" folder on dropbox so that your friends can add from that into their library for putting onto their devices and vice versa.

I wish I had like minded friends to do this myself, I spend all my time writing plugins etc nowadays so the cleanup of my own library progresses at a snails pace. Good luck and hope this clarifies things. I think it is less restricting and safer than using lockfiles, particularly if you friends are in the same timezone.

[kacir]

Here is another idea.
Each of users has his own local copy of Calibre Library, with metadata.db.
Setup some kind of CVS system - what programmers use when many people have to work on the same bunch of source code files. It can be CVS, mercurial, git, whatever. http://en.wikipedia.org/wiki/Compari...ntrol_software

You want to work on database maintenance. Before starting Calibre on your machine you import changed books and opf files from server, merging it with your database. Then you run Calibre repair database, to make your metadata.db reflect new version of database from server. When you finish the edit, you commit your changes (again without metadata.db) back to the server. You have to manually review any cases when the system finds file on server that has been changed since you checked out the files.

Such systems for developers are built to handle situations when several people make change to same source file.
As an added benefit you can return to state of your library from any given date.