Thoughts on implementing a networked Calibre database

[haertig]

Will Calibre work with an overlay file system (e.g., OverlayFS in Linux)?

I would like to share my Calibre database read-only over a network. I know, "Bad, bad, bad - you're on your own down the Highway To Hell, you idiot" will be the standard response I expect to hear. But read what I'm proposing.

What I'm thinking is:

(1) Calibre database on Computer A

(2) Network share of Computer A's database to Computer B, mounted read-only on Computer B

(3) OverlayFS implemented on Computer B - the "lower layer" being the read-only remote mount from Computer A, the "upper layer" being a read-write directory on Computer B (a local directory)

(4) Run Calibre on Computer B using the overlayed file system as its database

The above assumes that OverlayFS can successfully use a remote mount as the "lower layer", and a local directory as the "upper layer", of an overlay file system. I have never tried this, but until I get a chance to research it further I am assuming that it will work.

It appears that this would protect the main database (on Computer A) from corruption due to Calibre activity on Computer B, since the main database is read-only on Computer B.

I also appears that Calibre on Computer B should not balk at a read-only database because it is actually using a read-write layer on top of the actual database.

If Calibre on Computer B corrupted anything, it should theoretically corrupt only the local read-write layer. To fix any corruption, just stop Calibre on Computer B, tear down the overlay file system, delete the read-write layer directory, recreate the overlay file system, and restart Calibre. This is fine, because we really only wanted Calibre on Computer B to have read-only access to its database. Anything it tried to write, or corrupted by accident, can be safely discarded.

But hopefully Calibre on Computer B would not corrupt anything, because it's kind of on an island all by itself being the sole player with it's local read-write layer. That is, unless Calibre has some inherent problem with using an overlay file system.

Does this overlay approach have any chance of success as a kludgy solution to the "shared read-only remote database" use case for Calibre?

[kovidgoyal]

If your overlay filesystem actually implements correct filesystem semantics (which network filesystems almost never do) then sure it will work. If it doesn't, it wont.

[haertig]

I do not know, and can only assume, why Calibre does not work well with a networked database:

(1) There's the multi-user issue: two different instances of Calibre running on two different computers, both writing to the same database potentially located on a third computer. That kind of operation is a tough nut to crack and requires a transactional database designed for that kind of access. Calibre is a single user application, trying to make it multi-user without the required database support (which I assume it doesn't have) is just not going to work reliably. It's non-trivial to run multi-user on the SAME computer, try doing that from different computers over a network and things can get ugly fast.

(2) In theory, you might be able to limit things to only ONE Calibre instance being able to write to the database, and all other instances just have read-only access. This could probably work for the most part, with the chance of an occasional glitch if a Calibre instance was trying to read something that was in an inconsistent (unfinished write) state. Without the required database support as mentioned above, the read-only instance would not know that what it was trying to read was in an unstable state. But as I understand things from searching the web, Calibre does not like having just read-only access to its data. So it's a moot point talking about giving it a read-only database. Apparently it complains of corruption if it cannot write to its database. And either it starts writing at startup (maybe updates a timestamp or something), or it checks for write permission at startup. From what I read on the internet, it complains of database corruption immediately, before you've initiated any obvious transactions that would require writing.

I am not sure what Kovid meant by "implements correct filesystem semantics". Are we talking about file locking?

I can understand difficulties when using, say, an NTFS filesystem from a Windows box mounted to a Linux box via Samba. On the Linux end you can do something like "chmod 755" and the command will apparently succeed (per the Linux command line exit value), but the native NTFS filesystem doesn't know what to do with that command - the permissions structure on Windows is different than Linux - so you really didn't accomplish the chmod even though it might initially appear that you did from Linux's viewpoint. I would not expect Calibre to work 100% in a networked database situation like that.

But what about a Linux box hosting the Calibre database on an EXT3 partition that is mounted via NFS to another Linux box running Calibre? Would Calibre NOT work in that networked environment (ignoring the multi-user problem for the moment)?

This all probably comes down to "incorrect implementation of filesystem semantics", which I don't understand from a practical viewpoint. What is necessary, and how is it incorrectly implemented in a network mount?

I would like to try my overlay filesystem proposal, but I have no way of doing the required high load testing to really test if it works. Most likely, it WOULD work for me during testing, but that could be because of dumb luck in a light load situation. Just like all the people who claim that Calibre works fine with a network mount for them. Maybe it does, or maybe they just haven't run into it failing yet. But they will later. That's the situation you want to stay out of.

[kovidgoyal]

filesystem semantics are complicated. Which operations are atomic, what ordering guarantees there are, how fsync works, etc, etc. Network filesystems never get all of them right, in my experience. Trying to do so would make them unusably slow. Whether any particular tradeoff any particular implementation makes will break something calibre relies on, there is no a-priori way to know, you have to test.

And of course, if you have multiple things making modifications to the same library database/files, all bets are off, even without file system implementation issues.

[ownedbycats]

I saw a mention of hardlinking in the FAQ section about using a NAS:

A filesystem is a complex beast. Most network filesystems lack various filesystem features that calibre uses. Some don’t support file locking, some don’t support hardlinking, some are just flaky.

Is hardlinking something Calibre uses, or just an example of how filesystems differ?

[kovidgoyal]

Originally Posted by ownedbycats

I saw a mention of hardlinking in the FAQ section about using a NAS:



Is hardlinking something Calibre uses, or just an example of how filesystems differ?

Yes hardlinking is used to make moving files fast and robust alongside a corresponding database transaction.