Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Related Tools

Notices

Reply
 
Thread Tools Search this Thread
Old 04-21-2011, 06:21 AM   #1
rjalex
Member
rjalex began at the beginning.
 
Posts: 17
Karma: 10
Join Date: Apr 2011
Device: kindle 3
File lock for multiuser Calibre ???

We are a small group of friends that want to mantain a common Calibre library on cloud storage (DropBox).

Calibre would be installed on each of our laptops pointing to the local Dropbox folder.

Each of us can import a book to our library using his Calibre instance.

Each of us can correct any book metadata, add cover images, add tags.

Do you have suggestions on how to handle file locking so that ?

Thank you very much.
rjalex is offline   Reply With Quote
Old 04-21-2011, 06:48 AM   #2
theducks
Grand Sorcerer
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 14,633
Karma: 5628865
Join Date: Aug 2009
Location: (The original) Silicon Valley, USA
Device: Galaxy Tab 2, Astak Pocket Pro, K4NT
Quote:
Originally Posted by rjalex View Post
We are a small group of friends that want to mantain a common Calibre library on cloud storage (DropBox).

Calibre would be installed on each of our laptops pointing to the local Dropbox folder.

Each of us can import a book to our library using his Calibre instance.

Each of us can correct any book metadata, add cover images, add tags.

Do you have suggestions on how to handle file locking so that ?

Thank you very much.
Rewrite Calibre to be Multi-User while leaving it available for (novice) single user systems.

Calibre is Single User at a time


The 'Content Server' allows multiple read access.

If you want Write access, you might consider making a dedicated 'Calibre PC' available for Remote Desktop sharing (with gobs of access security) that allows a single-at-a-time logon. (This PC needs some CPU/Memory 'chops' as it will be doing all the work.)
theducks is offline   Reply With Quote
Old 04-21-2011, 07:08 AM   #3
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,224
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
If you genuinely need simultaneous write then as theducks has said your options are limited.

However what you "could" do is just treat dropbox as an add source/save to disk target. So you save to disk out of Calibre onto the DropBox location (making sure you save the opf/jpg). Then others can use Add from Folder, with automerge turned on. So that way you will pickup any metadata changes/new covers/versions etc others put in.

The issue with that is if the author/title is incorrect and gets changed so you get duplicates. So what I would do is have two folder areas - one for the "unprocessed" books and one for the "clean" ones. So you only ever import into your local library from the "clean" area to prevent duplicate issues from renaming.

Then the workflow becomes someone picks a folder of books from the "unprocessed" area and removes it from there to their local disk (so others dont try to clean them at the same time). Then import from that folder to Calibre, clean them up and then save to disk back into the "clean" folder in Dropbox for others to import, either just that author if you are communicating with each other or the whole clean folder if you use the automerge approach.

Sounds a nice way to share the burden of getting a clean library put together.
kiwidude is offline   Reply With Quote
Old 04-21-2011, 07:29 AM   #4
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,838
Karma: 12535517
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
To the OP --- tread carefully.

Quote:
Originally Posted by kiwidude View Post
However what you "could" do is just treat dropbox as an add source/save to disk target. So you save to disk out of Calibre onto the DropBox location (making sure you save the opf/jpg). Then others can use Add from Folder, with automerge turned on. So that way you will pickup any metadata changes/new covers/versions etc others put in.
....
Then the workflow becomes someone picks a folder of books from the "unprocessed" area and removes it from there to their local disk (so others dont try to clean them at the same time). Then import from that folder to Calibre, clean them up and then save to disk back into the "clean" folder in Dropbox for others to import, either just that author if you are communicating with each other or the whole clean folder if you use the automerge approach.
The dropbox is a local folder and each of the three machines. they can each be working on the library at the same time. Even if they were working on different books when each of their metadata.db files tries to update the main cloud (and in turn each of their local machines) it is very probable that they will experience blocked files.

Using dropbox in this manner is only a viable option if they work out some sort of time schedule. If they ever update close to the same time they will be frustrated with missing books that get locked as conflicted copies on their local machines.
DoctorOhh is offline   Reply With Quote
Old 04-21-2011, 07:31 AM   #5
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,224
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
@dwanthny - you misunderstand me I think. They would have their own Calibre library that is not in dropbox - the metadata.db is not shared. As I said above they just use Dropbox to Add from as a source folder into their local library. No concurrency issues whatsoever.
kiwidude is offline   Reply With Quote
Old 04-21-2011, 07:34 AM   #6
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,838
Karma: 12535517
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Quote:
Originally Posted by kiwidude View Post
@dwanthny - you misunderstand me I think. They would have their own Calibre library that is not in dropbox - the metadata.db is not shared. As I said above they just use Dropbox to Add from as a source folder into their local library. No concurrency issues whatsoever.
I understood you. Do you use Dropbox? If they all decided to copy to their dropbox (local machine) at or close to the same time then it is probable that they will cause the metadata.db file in the dropbox to get conflicted and not update the cloud or another's local copy of the dropbox metadata.db file.

Update: its not a calibre concurrency issue it is a dropbox concurrency issue.

Last edited by DoctorOhh; 04-21-2011 at 07:36 AM.
DoctorOhh is offline   Reply With Quote
Old 04-21-2011, 07:36 AM   #7
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,224
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
Yes I do use dropbox. There is no metadata.db file stored there with what I propose.

They have a local Calibre library on their machine, not within a dropbox folder. The dropbox folders are populated using save to disk exporting from their library - so dropbox will contain opf/jpg and book files, but no metadata.db.
kiwidude is offline   Reply With Quote
Old 04-21-2011, 07:39 AM   #8
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,838
Karma: 12535517
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Quote:
Originally Posted by kiwidude View Post
Yes I do use dropbox. There is no metadata.db file stored there with what I propose.

They have a local Calibre library on their machine, not within a dropbox folder. The dropbox folders are populated using save to disk exporting from their library - so dropbox will contain opf/jpg and book files, but no metadata.db.

Sounds like a plan.

I guess its time for me to go to bed.
DoctorOhh is offline   Reply With Quote
Old 04-21-2011, 10:22 AM   #9
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,931
Karma: 5036099
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
I would suggest the following trick. Write a small batch file/script to launch calibre. Have the batch file check for the existence of a dummy file in the dropbox folder if it exists it should abort, if it doesn't it should create it and launch calibre. Once calibre quits it should delete the file.

Note that this isn't fully robust since the file isn't a proper lock file, but it might be "good enough" if only a small number of people are doing this.

Backup your calibre library often.
kovidgoyal is offline   Reply With Quote
Old 04-22-2011, 03:01 AM   #10
rjalex
Member
rjalex began at the beginning.
 
Posts: 17
Karma: 10
Join Date: Apr 2011
Device: kindle 3
Thank you all.

Yes I had thought about the "manual lock file" idea. We are three friends and we would impose the discipline to write a "namemmdd.lock" file (e.g. "bob0422.lock") on dropbox and delete it when we finish using it, and the batch file could be a great idea to add. The purpose of writing name/date is that if one of us "forgets" the lock there we can ring him/her up ask as to why and then carry on

Also the "dropbox as data vault out of calibre's jurisdiction with a "clean" and "to fix" directories" is a clever idea. If I get your idea well the "to fix" data would be "added" to a non-dropbox local librabry adn when clean exported "to disk" where the disk is the "clean" book with the mobi, opf etc files ? The only part I am not quite understanding is then how to use cailbre to load the clean books on my kindle (very new user hence I might be unaware of Calibre being able to use two different libraries).

Thanks to all and happy Easter.
Bob
rjalex is offline   Reply With Quote
Old 04-22-2011, 04:20 AM   #11
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,224
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
Quote:
Originally Posted by rjalex View Post
Also the "dropbox as data vault out of calibre's jurisdiction with a "clean" and "to fix" directories" is a clever idea. If I get your idea well the "to fix" data would be "added" to a non-dropbox local librabry adn when clean exported "to disk" where the disk is the "clean" book with the mobi, opf etc files ? The only part I am not quite understanding is then how to use cailbre to load the clean books on my kindle (very new user hence I might be unaware of Calibre being able to use two different libraries).
Whether you have two local libraries or one is up to you and whether you want to mix the batch of books you are working on cleaning up currently with the books that you have cleaned so far. In terms of getting books onto your kindle it would be as per usual of doing send to device from your local library. You would likely not want any unclean books on there. So either you could use a tag or custom column when you add books from the "to fix" folder to your single local library which you remove as you clean them up. Or you could have two local libraries, one for your to fix books and one for your clean ones. It is trivial with calibre to switch libraries and to move books between them.

So the choice is up to you and each of you friends can make their own decision as the libraries are local and not shared. The important thing is just the save to disk back into the "clean" folder on dropbox so that your friends can add from that into their library for putting onto their devices and vice versa.

I wish I had like minded friends to do this myself, I spend all my time writing plugins etc nowadays so the cleanup of my own library progresses at a snails pace. Good luck and hope this clarifies things. I think it is less restricting and safer than using lockfiles, particularly if you friends are in the same timezone.
kiwidude is offline   Reply With Quote
Old 04-22-2011, 04:43 AM   #12
kacir
Wizard
kacir ought to be getting tired of karma fortunes by now.kacir ought to be getting tired of karma fortunes by now.kacir ought to be getting tired of karma fortunes by now.kacir ought to be getting tired of karma fortunes by now.kacir ought to be getting tired of karma fortunes by now.kacir ought to be getting tired of karma fortunes by now.kacir ought to be getting tired of karma fortunes by now.kacir ought to be getting tired of karma fortunes by now.kacir ought to be getting tired of karma fortunes by now.kacir ought to be getting tired of karma fortunes by now.kacir ought to be getting tired of karma fortunes by now.
 
kacir's Avatar
 
Posts: 2,788
Karma: 3098803
Join Date: May 2006
Device: PocketBook 360, before it was Sony Reader, cassiopeia A-20
Here is another idea.
Each of users has his own local copy of Calibre Library, with metadata.db.
Setup some kind of CVS system - what programmers use when many people have to work on the same bunch of source code files. It can be CVS, mercurial, git, whatever. http://en.wikipedia.org/wiki/Compari...ntrol_software

You want to work on database maintenance. Before starting Calibre on your machine you import changed books and opf files from server, merging it with your database. Then you run Calibre repair database, to make your metadata.db reflect new version of database from server. When you finish the edit, you commit your changes (again without metadata.db) back to the server. You have to manually review any cases when the system finds file on server that has been changed since you checked out the files.

Such systems for developers are built to handle situations when several people make change to same source file.
As an added benefit you can return to state of your library from any given date.
kacir is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
File Replacement outside of Calibre Zipr Calibre 1 02-13-2011 06:34 PM
Calibre debug file meme Calibre 7 02-01-2011 04:45 PM
Calibre File Names ll Legion ll Calibre 4 10-13-2010 05:03 PM
calibre 0.6.17 can't convert any file Phil_C Calibre 9 10-11-2009 09:02 PM
File Names in Calibre jimbo Calibre 6 07-19-2008 09:21 PM


All times are GMT -4. The time now is 10:08 PM.


MobileRead.com is a privately owned, operated and funded community.