MobileRead Forums - View Single Post - Thoughts on implementing a networked Calibre database

haertig · 01-04-2021, 03:04 AM

I do not know, and can only assume, why Calibre does not work well with a networked database:

(1) There's the multi-user issue: two different instances of Calibre running on two different computers, both writing to the same database potentially located on a third computer. That kind of operation is a tough nut to crack and requires a transactional database designed for that kind of access. Calibre is a single user application, trying to make it multi-user without the required database support (which I assume it doesn't have) is just not going to work reliably. It's non-trivial to run multi-user on the SAME computer, try doing that from different computers over a network and things can get ugly fast.

(2) In theory, you might be able to limit things to only ONE Calibre instance being able to write to the database, and all other instances just have read-only access. This could probably work for the most part, with the chance of an occasional glitch if a Calibre instance was trying to read something that was in an inconsistent (unfinished write) state. Without the required database support as mentioned above, the read-only instance would not know that what it was trying to read was in an unstable state. But as I understand things from searching the web, Calibre does not like having just read-only access to its data. So it's a moot point talking about giving it a read-only database. Apparently it complains of corruption if it cannot write to its database. And either it starts writing at startup (maybe updates a timestamp or something), or it checks for write permission at startup. From what I read on the internet, it complains of database corruption immediately, before you've initiated any obvious transactions that would require writing.

I am not sure what Kovid meant by "implements correct filesystem semantics". Are we talking about file locking?

I can understand difficulties when using, say, an NTFS filesystem from a Windows box mounted to a Linux box via Samba. On the Linux end you can do something like "chmod 755" and the command will apparently succeed (per the Linux command line exit value), but the native NTFS filesystem doesn't know what to do with that command - the permissions structure on Windows is different than Linux - so you really didn't accomplish the chmod even though it might initially appear that you did from Linux's viewpoint. I would not expect Calibre to work 100% in a networked database situation like that.

But what about a Linux box hosting the Calibre database on an EXT3 partition that is mounted via NFS to another Linux box running Calibre? Would Calibre NOT work in that networked environment (ignoring the multi-user problem for the moment)?

This all probably comes down to "incorrect implementation of filesystem semantics", which I don't understand from a practical viewpoint. What is necessary, and how is it incorrectly implemented in a network mount?

I would like to try my overlay filesystem proposal, but I have no way of doing the required high load testing to really test if it works. Most likely, it WOULD work for me during testing, but that could be because of dumb luck in a light load situation. Just like all the people who claim that Calibre works fine with a network mount for them. Maybe it does, or maybe they just haven't run into it failing yet. But they will later. That's the situation you want to stay out of.

01-04-2021, 03:04 AM	#3
haertig Wizard Posts: 1,899 Karma: 31522252 Join Date: Sep 2017 Device: PW3, Fire HD8 Gen7, Moto G7, Sansa Clip v2, Ruizu X26	I do not know, and can only assume, why Calibre does not work well with a networked database: (1) There's the multi-user issue: two different instances of Calibre running on two different computers, both writing to the same database potentially located on a third computer. That kind of operation is a tough nut to crack and requires a transactional database designed for that kind of access. Calibre is a single user application, trying to make it multi-user without the required database support (which I assume it doesn't have) is just not going to work reliably. It's non-trivial to run multi-user on the SAME computer, try doing that from different computers over a network and things can get ugly fast. (2) In theory, you might be able to limit things to only ONE Calibre instance being able to write to the database, and all other instances just have read-only access. This could probably work for the most part, with the chance of an occasional glitch if a Calibre instance was trying to read something that was in an inconsistent (unfinished write) state. Without the required database support as mentioned above, the read-only instance would not know that what it was trying to read was in an unstable state. But as I understand things from searching the web, Calibre does not like having just read-only access to its data. So it's a moot point talking about giving it a read-only database. Apparently it complains of corruption if it cannot write to its database. And either it starts writing at startup (maybe updates a timestamp or something), or it checks for write permission at startup. From what I read on the internet, it complains of database corruption immediately, before you've initiated any obvious transactions that would require writing. I am not sure what Kovid meant by "implements correct filesystem semantics". Are we talking about file locking? I can understand difficulties when using, say, an NTFS filesystem from a Windows box mounted to a Linux box via Samba. On the Linux end you can do something like "chmod 755" and the command will apparently succeed (per the Linux command line exit value), but the native NTFS filesystem doesn't know what to do with that command - the permissions structure on Windows is different than Linux - so you really didn't accomplish the chmod even though it might initially appear that you did from Linux's viewpoint. I would not expect Calibre to work 100% in a networked database situation like that. But what about a Linux box hosting the Calibre database on an EXT3 partition that is mounted via NFS to another Linux box running Calibre? Would Calibre NOT work in that networked environment (ignoring the multi-user problem for the moment)? This all probably comes down to "incorrect implementation of filesystem semantics", which I don't understand from a practical viewpoint. What is necessary, and how is it incorrectly implemented in a network mount? I would like to try my overlay filesystem proposal, but I have no way of doing the required high load testing to really test if it works. Most likely, it WOULD work for me during testing, but that could be because of dumb luck in a light load situation. Just like all the people who claim that Calibre works fine with a network mount for them. Maybe it does, or maybe they just haven't run into it failing yet. But they will later. That's the situation you want to stay out of.