Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Development

Notices

Reply
 
Thread Tools Search this Thread
Old 03-15-2018, 06:32 PM   #1
aurne
Junior Member
aurne began at the beginning.
 
Posts: 4
Karma: 10
Join Date: Mar 2018
Device: kindle
What are the reasons calibre can't support multi-user libraries?

I'm planning on implementing better networked library support for calibre, essentially allowing multiple clients to connect to a centralized library.

As sqlite supports multiple client connections, albeit locks the whole database during writes, I'm wondering why multiple clients currently cause database corruptions and what code interfaces with the db (is it all in src/db/ or is there more).

Currently considering a client/server model where the server is the sole writer and processes db IO requests sent by clients.

Please let me know your thoughts, and if there is anything important I'm missing.
aurne is offline   Reply With Quote
Old 03-15-2018, 07:41 PM   #2
ilovejedd
hopeless n00b
ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.
 
ilovejedd's Avatar
 
Posts: 5,111
Karma: 19597086
Join Date: Jan 2009
Location: in the middle of nowhere
Device: PW4, PW3, Libra H2O, iPad 10.5, iPad 11, iPad 12.9
I believe multiple users are already supported via the content server.
ilovejedd is offline   Reply With Quote
Advert
Old 03-15-2018, 07:59 PM   #3
aurne
Junior Member
aurne began at the beginning.
 
Posts: 4
Karma: 10
Join Date: Mar 2018
Device: kindle
Quote:
Originally Posted by ilovejedd View Post
I believe multiple users are already supported via the content server.
Unfortunately the content server is severely limited in terms of functionality such as editing book metadata (and thus database operations), compared to the desktop version.
aurne is offline   Reply With Quote
Old 03-15-2018, 11:32 PM   #4
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 20,459
Karma: 26645808
Join Date: Mar 2012
Location: Sydney Australia
Device: none
@aurne - I believe one reason will be that the 'database' on which desktop calibre operates is not the on-disk metadata.db sqlite database as such. Instead it operates on an in-memory 'database' which is initially built from the on-disk sqlite metadata.db when calibre starts. See API documentation for the database interface

BR
BetterRed is online now   Reply With Quote
Old 03-15-2018, 11:46 PM   #5
aurne
Junior Member
aurne began at the beginning.
 
Posts: 4
Karma: 10
Join Date: Mar 2018
Device: kindle
Quote:
Originally Posted by BetterRed View Post
@aurne - I believe one reason will be that the 'database' on which desktop calibre operates is not the on-disk metadata.db sqlite database as such. Instead it operates on an in-memory 'database' which is initially built from the on-disk sqlite metadata.db when calibre starts. See API documentation for the database interface

BR
Thanks! That's pretty informative, though it seems like a really non-standard way to handle databases. The wiki quotes "maximum performance" as justification. Do you, or anyone else have any numbers regarding the performance differential between in-memory and traditional sqlite?

Anyways, it seems like my original architecture of a single server performing db IO with multiple clients connected will be the best choice going forward.
aurne is offline   Reply With Quote
Advert
Old 03-16-2018, 12:14 AM   #6
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 20,459
Karma: 26645808
Join Date: Mar 2012
Location: Sydney Australia
Device: none
@aurne - I'm sure Kovid will be along soon, and I've no doubt he'll give you a definitive response. It's an issue he has commented on at various times, sorry I don't have a link to hand. A search for client-server in the calibre forum might find something.

See also - Kovid's TODO list

BR
BetterRed is online now   Reply With Quote
Old 03-16-2018, 07:46 AM   #7
DaltonST
Deviser
DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.
 
DaltonST's Avatar
 
Posts: 2,265
Karma: 2090983
Join Date: Aug 2013
Location: Texas
Device: none
Quote:
Do you, or anyone else have any numbers regarding the performance differential between in-memory and traditional sqlite?
Aurne:

As a proxy answer, search for the reason that people use SSDs instead of Hard Drives.


DaltonST
DaltonST is offline   Reply With Quote
Old 03-16-2018, 11:47 AM   #8
DNSB
Bibliophagist
DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.
 
DNSB's Avatar
 
Posts: 34,557
Karma: 144552660
Join Date: Jul 2010
Location: Vancouver
Device: Kobo Sage, Forma, Clara HD, Lenovo M8 FHD, Paperwhite 4, Tolino epos
Quote:
Originally Posted by aurne View Post
Thanks! That's pretty informative, though it seems like a really non-standard way to handle databases. The wiki quotes "maximum performance" as justification. Do you, or anyone else have any numbers regarding the performance differential between in-memory and traditional sqlite?
Not sure how relevant to sqlite this is but Microsoft's SQL 2014 and 2016 have the capability to use in-memory OLTP. Various tests showed a 30x increase in performance depending on what your task mix is. You just need lots of memory to hold the database -- in our case, the SQL servers have 1TB of memory with a flash SAN for disk storage.
DNSB is offline   Reply With Quote
Old 03-16-2018, 11:58 AM   #9
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,778
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
The calibre database is perfectly multi-user, the calibre GUI is not. And the as of the next calibre release the content server will support editing metadata, which you can also already do using calibredb with a remote database.
kovidgoyal is offline   Reply With Quote
Old 03-16-2018, 02:12 PM   #10
sjfan
Addict
sjfan ought to be getting tired of karma fortunes by now.sjfan ought to be getting tired of karma fortunes by now.sjfan ought to be getting tired of karma fortunes by now.sjfan ought to be getting tired of karma fortunes by now.sjfan ought to be getting tired of karma fortunes by now.sjfan ought to be getting tired of karma fortunes by now.sjfan ought to be getting tired of karma fortunes by now.sjfan ought to be getting tired of karma fortunes by now.sjfan ought to be getting tired of karma fortunes by now.sjfan ought to be getting tired of karma fortunes by now.sjfan ought to be getting tired of karma fortunes by now.
 
Posts: 281
Karma: 7724454
Join Date: Sep 2017
Location: Bethesda, MD, USA
Device: Kobo Aura H20, Kobo Clara HD
Quote:
Originally Posted by kovidgoyal View Post
The calibre database is perfectly multi-user, the calibre GUI is not. And the as of the next calibre release the content server will support editing metadata, which you can also already do using calibredb with a remote database.
Just for clarity, are you saying it's safe to run calibredb against the same database while a GUI (or content server) is running on that database?
sjfan is offline   Reply With Quote
Old 03-16-2018, 02:19 PM   #11
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,778
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
calibredb can talk to the calibre content server, wherever it is running, either standalone or inside a calibre GUI. In either case it is safe to do so, with the obvious caveat that if you do something like deleting a book while simultaneously editing metadata for the same book, you will get an error for one of the two operations, but the other one will succeed.
kovidgoyal is offline   Reply With Quote
Old 03-16-2018, 06:38 PM   #12
aurne
Junior Member
aurne began at the beginning.
 
Posts: 4
Karma: 10
Join Date: Mar 2018
Device: kindle
Quote:
Originally Posted by kovidgoyal View Post
The calibre database is perfectly multi-user, the calibre GUI is not. And the as of the next calibre release the content server will support editing metadata, which you can also already do using calibredb with a remote database.
The change I am proposing is to allow the calibre GUI to interface with a remote database, possibly through calibredb, as the desktop GUI is quite powerful and intuitive.

Just to confirm, so I don't duplicate planned functionality, this is not the feature that will be present in the next release?
aurne is offline   Reply With Quote
Old 03-17-2018, 01:45 AM   #13
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,778
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Quote:
Originally Posted by aurne View Post
The change I am proposing is to allow the calibre GUI to interface with a remote database, possibly through calibredb, as the desktop GUI is quite powerful and intuitive.

Just to confirm, so I don't duplicate planned functionality, this is not the feature that will be present in the next release?
No, it is not.
kovidgoyal is offline   Reply With Quote
Old 04-22-2018, 12:18 PM   #14
eschwartz
Ex-Helpdesk Junkie
eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.
 
eschwartz's Avatar
 
Posts: 19,422
Karma: 85397180
Join Date: Nov 2012
Location: The Beaten Path, USA, Roundworld, This Side of Infinity
Device: Kindle Touch fw5.3.7 (Wifi only)
Quote:
Originally Posted by aurne View Post
The change I am proposing is to allow the calibre GUI to interface with a remote database, possibly through calibredb, as the desktop GUI is quite powerful and intuitive.

Just to confirm, so I don't duplicate planned functionality, this is not the feature that will be present in the next release?
It is in fact what is referenced in the TODO which BetterRed linked, as:

Quote:
(2) is preferable in terms of features/functionality, but is much harder to implement in a performant and semantically correct manner.
As far as I am aware, Kovid has no current plans implementing either one, hence why it is dropped unceremoniously into the TODO until he can figure out what to do with them.
It's a bit of a challenging issue as indicated... if you can implement this I'm sure lots of people will be quite interested.
eschwartz is offline   Reply With Quote
Reply

Tags
database, development, multi-user, server, sqlite3

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
HarperCollins to Offer Multi-User E-Book Access to Libraries tubemonkey News 11 07-07-2017 09:32 PM
Multi-user calibre library malamona Development 3 07-11-2013 11:48 AM
Google Play store & multi-user support avantman42 Kindle Fire 11 04-11-2013 08:17 PM
Multi language support via Calibre Jim77 Calibre 4 02-12-2010 02:46 PM


All times are GMT -4. The time now is 03:31 AM.


MobileRead.com is a privately owned, operated and funded community.