Feature request: Calibre Multi user/device access

Cool Javelin · 06-08-2018, 02:42 PM

First, right off the bat, I just like to say Calibre is the greatest thing since sliced bread. It is mature, stable, and does whet it is designed to do extremely well. Better in fact then most other programs I use.

Kovid.

That said:

My guess is this has been asked from time to time but I'd just like to add another brick on the wall.

It would be nice if the Calibre database could be accessed by more then one user/computer at a time.

I have my database on one of my slowest computers as the server seems to perform OK there, and the web access is slow anyway. But I like to edit the database (via Calibre proper) from my fastest computer which is the one I generally sit at.

I understand this is bad to alter the database via the network, so I have been using a remote desktop to run Calibre on my slow machine and that is, well, slow.

I think multi user access could be implemented by having a Database Server Engine (DSE) on the computer with the database, and all other programs (Calibre, Content Server, etc...) alter the db via the DSE similar to the way Quickbooks handles multi user access.

You could use sockets, or maybe another IPC (inter process communication) method. Basically, change all reads/writes from/to the database to socket requests and the single DSE handles them in some kind of order.

This would eliminate the issue of one user altering the database without another user knowing about it and corrupting the database.

Just my 2 cents.

Thanks for listening,
Mark.

BetterRed · 06-08-2018, 07:29 PM

FYI - https://github.com/kovidgoyal/calibr...e/srv/TODO.rst

kovidgoyal · 06-08-2018, 09:30 PM

It isn't that simple, the problem is not just concurrent access to the db, the problem is that when client A changes something, all other clients have to somehow update their views of that something as well. To do so in a manner that is both acceptably performant and semantically correct is very far from trivial.

Adoby · 06-09-2018, 08:19 AM

One way to handle the use case in the first post, server on one computer, update on another, is to simply copy the calibre library. After editing/adding books on the fast computer, copy the whole library over to the server, replacing the old library. Possibly using a sync app to make it faster.

That way you can access the calibre library from both computers and have a backup as well.

Treat the backup on the server as a read-only copy.

You could have two copies on the server to avoid writing to the calibre library when the server is running. One copy that the server is using. One copy that is updated from the fast computer. And update th copy the server is using when the server is (re-)started. Perhaps using some scheduler? Or a "trigger file" added when the backup is updated.

Cool Javelin · 06-21-2018, 05:29 PM

Adoby:

I tried your idea (copy lib, edit lib, stop server, copy back, start server) and it works as expected. My lib is about 5GB, so it takes a while to copy.

Of course, I had to rename my old original lib to something else, then copy the new so folders that got changed didn't create duplicates.

It occurred to me just now (while writing this) I could use a syncing tool to only update the changes. (Make the changes, stop the content server, sync the folders, then restart the server.)

Great idea.

Kovid:

I do have an understanding of, and some experience writing, multi user access to database and saying it isn't trivial is a bit of an understatement.

Getting the clients to update in real time is the best approach but also the most complex. Simpler is to detect a change to the database and require the user to reload it but that isn't as much fun.

Calibre is very mature and quite stable. Most new features will become more and more complex as the simpler bugs get worked out.

I was just thinking next time you are bored, you know, when you have a few weeks of free time on your hands and don't have to make a living or anything like that. (Damn, it sure is easy to ask for a big change to a free program.)

Mark.

Adoby · 06-22-2018, 07:02 AM

It could be a fun little project to automate this.

On the workstation:

Add some commands to the script that start calibre to also update a copy of the library on the server, when you exit calibre. Could use a sync app over the network.

On the server:

Run a script every 10 minutes or so. Check if a new update has arrived. If so, turn off the calibre server and update the library being served. Start the calibre server again.

You may need some semaphore files, some atomic renames and perhaps some extra copies of the library to make sure it is clean and safe.

06-08-2018, 02:42 PM	#1
Cool Javelin Enthusiast Posts: 28 Karma: 18222 Join Date: Aug 2012 Device: Kindle (early)	Feature request: Calibre Multi user/device access First, right off the bat, I just like to say Calibre is the greatest thing since sliced bread. It is mature, stable, and does whet it is designed to do extremely well. Better in fact then most other programs I use. Kovid. That said: My guess is this has been asked from time to time but I'd just like to add another brick on the wall. It would be nice if the Calibre database could be accessed by more then one user/computer at a time. I have my database on one of my slowest computers as the server seems to perform OK there, and the web access is slow anyway. But I like to edit the database (via Calibre proper) from my fastest computer which is the one I generally sit at. I understand this is bad to alter the database via the network, so I have been using a remote desktop to run Calibre on my slow machine and that is, well, slow. I think multi user access could be implemented by having a Database Server Engine (DSE) on the computer with the database, and all other programs (Calibre, Content Server, etc...) alter the db via the DSE similar to the way Quickbooks handles multi user access. You could use sockets, or maybe another IPC (inter process communication) method. Basically, change all reads/writes from/to the database to socket requests and the single DSE handles them in some kind of order. This would eliminate the issue of one user altering the database without another user knowing about it and corrupting the database. Just my 2 cents. Thanks for listening, Mark.

06-09-2018, 08:19 AM	#4
Adoby Handy Elephant Posts: 1,736 Karma: 26785668 Join Date: Dec 2009 Location: Southern Sweden, far out in the quiet woods Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus	One way to handle the use case in the first post, server on one computer, update on another, is to simply copy the calibre library. After editing/adding books on the fast computer, copy the whole library over to the server, replacing the old library. Possibly using a sync app to make it faster. That way you can access the calibre library from both computers and have a backup as well. Treat the backup on the server as a read-only copy. You could have two copies on the server to avoid writing to the calibre library when the server is running. One copy that the server is using. One copy that is updated from the fast computer. And update th copy the server is using when the server is (re-)started. Perhaps using some scheduler? Or a "trigger file" added when the backup is updated. Last edited by Adoby; 06-09-2018 at 08:22 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
HarperCollins to Offer Multi-User E-Book Access to Libraries	tubemonkey	News	11	07-07-2017 09:32 PM
Feature Request: Multi-Language Spell Check	Tex2002ans	Sigil	3	07-14-2016 12:32 PM
Feature Request: Mask calibre server connection string with user configurable name	Zealousky	Calibre Companion	8	02-20-2015 08:17 AM
Multi-user access to Calibre db in cloud	higlider	Related Tools	2	12-05-2013 06:32 AM
Request for Ideas: Multi-user updates	mc3	Library Management	4	05-05-2012 10:15 AM

06-08-2018, 07:29 PM	#2
BetterRed null operator (he/him) Posts: 20,568 Karma: 26954694 Join Date: Mar 2012 Location: Sydney Australia Device: none	FYI - https://github.com/kovidgoyal/calibr...e/srv/TODO.rst

06-08-2018, 09:30 PM	#3
kovidgoyal creator of calibre Posts: 43,858 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	It isn't that simple, the problem is not just concurrent access to the db, the problem is that when client A changes something, all other clients have to somehow update their views of that something as well. To do so in a manner that is both acceptably performant and semantically correct is very far from trivial.

06-21-2018, 05:29 PM	#5
Cool Javelin Enthusiast Posts: 28 Karma: 18222 Join Date: Aug 2012 Device: Kindle (early)	Adoby: I tried your idea (copy lib, edit lib, stop server, copy back, start server) and it works as expected. My lib is about 5GB, so it takes a while to copy. Of course, I had to rename my old original lib to something else, then copy the new so folders that got changed didn't create duplicates. It occurred to me just now (while writing this) I could use a syncing tool to only update the changes. (Make the changes, stop the content server, sync the folders, then restart the server.) Great idea. Kovid: I do have an understanding of, and some experience writing, multi user access to database and saying it isn't trivial is a bit of an understatement. Getting the clients to update in real time is the best approach but also the most complex. Simpler is to detect a change to the database and require the user to reload it but that isn't as much fun. Calibre is very mature and quite stable. Most new features will become more and more complex as the simpler bugs get worked out. I was just thinking next time you are bored, you know, when you have a few weeks of free time on your hands and don't have to make a living or anything like that. (Damn, it sure is easy to ask for a big change to a free program.) Mark.

06-22-2018, 07:02 AM	#6
Adoby Handy Elephant Posts: 1,736 Karma: 26785668 Join Date: Dec 2009 Location: Southern Sweden, far out in the quiet woods Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus	It could be a fun little project to automate this. On the workstation: Add some commands to the script that start calibre to also update a copy of the library on the server, when you exit calibre. Could use a sync app over the network. On the server: Run a script every 10 minutes or so. Check if a new update has arrived. If so, turn off the calibre server and update the library being served. Start the calibre server again. You may need some semaphore files, some atomic renames and perhaps some extra copies of the library to make sure it is clean and safe.