Simple Approach to an External Library and Server using 3.14.0

[dlfuller]

I have the calibre app running on macOS with its library on a headless Ubuntu Server. The content server is never started from the menubar in the app.

Calibre is also installed on Ubuntu Server providing the calibredb command line interface to the calibre database and calibre-server which starts the server.

My simple approach with publication revisions is to modify them in the calibre app on the Mac then quit. Next stop and relaunch calibre-server on the Ubuntu Server.

sudo killall calibre-server
sudo calibre-server &

Those revised publications are then available to all devices on the server.

To add books, I add or drop them into the calibre app, check, modify, and convert if necessary. Quit the app. Again the same routine, stop and relaunch calibre-server on the Ubuntu Server.

sudo killall calibre-server
sudo calibre-server &

The publications are added to the library on the server and available to all devices.

The steps are easy to do and the calibre app keeps up to date. What might I be missing with this simple technique that would allow data corruption or loss?

[kovidgoyal]

The main concern with this is that macOS file systems and linux filesstems are different (one is case sensistive, one isn't) so directly sharing the same library between them can be problematic.

[dlfuller]

Thanks for the insight @kovidgoyal (and of course for all your ongoing efforts with calibre).

Is this a solvable situation?

As an example, what might happen the way it is? My use of the macOS calibre app is limited to adding books, format conversions, adding tags, revising metadata, etc. Once those publications are available on the server they are only read or downloaded through the calibre-server URL from multiple devices regardless of OS. Basically no reading in the macOS calibre app.

[JSWolf]

Given that the library is on Linux, use that and not OSX.

[dlfuller]

Fine @JSWolf, except this is a headless server. Those adding or editing activities for books are a pain just using the CLI.

[davidfor]

Originally Posted by dlfuller

Fine @JSWolf, except this is a headless server. Those adding or editing activities for books are a pain just using the CLI.

Calibre can be made to watch a directory and load any books found there. Try using the save-to-disk from the MAC and put them in a shared folder on the server to pickup. Or, save them on the MAC and use a script to process the folder running calibredb against each to send them to the server instance of calibre.

Actually, you could probably use the Open With plugin to add a menu item to call the calibredb command and send the book to the server. But, you would want to update the metadata and cover before doing this. The above method would do this as part of the save-to-disk.

[BetterRed]

Originally Posted by davidfor

Calibre can be made to watch a directory and load any books found there. Try using the save-to-disk from the MAC and put them in a shared folder on the server to pickup. Or, save them on the MAC and use a script to process the folder running calibredb against each to send them to the server instance of calibre.

Actually, you could probably use the Open With plugin to add a menu item to call the calibredb command and send the book to the server. But, you would want to update the metadata and cover before doing this. The above method would do this as part of the save-to-disk.

@davidfor - the first post has this

Originally Posted by dlfuller

I have the calibre app running on macOS with its library on a headless Ubuntu Server.

I interpret that as saying the Mac install of calibre is accessing a library on the server, which is presumably running Ext3 or Ext4. When calibre-server is run at the server it presumably uses the same library.

My question would be - will the files calibre creates and maintains from the MAC client, be named according to Ext3/4 rules (case sensitive) or HFS+ rules (case insensitive). I have no idea, so I would do some experiments

BR

[kovidgoyal]

It's on my TODO list to add support for adding books/converting them and editing metadata to the browser interface, I just did not have time to finish it for calibre 3.x and unfortunately because of some stuff in my personal life, my calibre development time is severely curtailed at the moment, hopefully it will pick up steam again soon and the server will gain those features.

[davidfor]

Originally Posted by BetterRed

@davidfor - the first post has this



I interpret that as saying the Mac install of calibre is accessing a library on the server, which is presumably running Ext3 or Ext4. When calibre-server is run at the server it presumably uses the same library.

I misread the first post. And based on the comments made since calibre v3 was released I'm surprised it works. The OP seems to have two instances of calibre (one for the server, one for the GUI) running at the same time on the same library. I'm surprise it is working. If the OP was stopping the server before opening the GUI, then the only concern would be the file systems.

My question would be - will the files calibre creates and maintains from the MAC client, be named according to Ext3/4 rules (case sensitive) or HFS+ rules (case insensitive). I have no idea, so I would do some experiments

Whether that is an issue depends on how the MAC is mounting the share from the server. It should be mounting it as a networked file system, not a native file system such as HFS+ or an EXT variant. I know Windows has no issues mounting EXT3/4 shares and using them. I can't check now, but I'm reasonably sure there is no way to tell what the remote file system is. It's probable that the client OS concerned won't support the full function of the underlying physical file system used on server, but the server should be returning errors to the client OS when it does something it shouldn't. The differences in case sensitivity mentioned by Kovid are a good example of this, and I don't remember how Windows handles this. If the server has files with the same name but different cases, I don't know if Windows will show in a directory listing. I have played with this in the past, but I don't remember exactly what happen, and I'm away from home and can't test what happens.

It will also be different depending on how the remote drive is accessed. Samba, NFS, FTP or something else will probably all give slightly different results.

[BetterRed]

@davidfor - and I might have misread a different part of the opening post

I thought it said the calibre-server was being stopped before starting the MAC client. But on re-reading it would appear the OP leaves calibre-server running while updating the library from the MAC, then when the MAC is closed the server is stopped and started. My guess is that this sequence is used to minimise the server unavailability time.

Would calibre-server.exe running on the server against a library on a local drive detect that a MAC elsewhere on the network was running the calibre GUI against the same library - or vice versa? I am under the impression it wouldn't, but I'm not up to speed with the version 3 server - I use the old 2.85.1 server.

I recall reading of user implemented schemes where the presence of a 'library.lock' file in the server hosted library folder was used to prevent concurrent execution of calibre.exe on two or more workstations.

BR

[davidfor]

My use of the calibre server has always been pretty basic. It's enabled to run when I have calibre running on my laptop. I really only use it as one of the tests for the browser on my Kobo devices.

I doubt if a service running on a Linux server will be able to tell if a client machine is directly accessing a file on the server. It depends on what locking is done. And that depends on the applications, OSes and file systems involved.

From what I understand of what Kovid has said about the v3 server, what the OP is doing shouldn't work. Or will have problems. The calibre UI is updating the database, independently of the calibre server, and there is a potential for disaster. What the OP is doing sounds like something for calibre v2 and they haven't changed.

And yes, lock files have been a popular way of trying to prevent multiple updates or uses of an application. They can work but everyone (code or people) has always update them appropriately. Otherwise, all you hear in the office is someone searching for someone to remove the lock file. Or trying to find the person who went to lunch without closing the application and now their PC is locked.

[wanni]

I'm running as well 2 clients against 1 DB. One on the server (windows) where the books and the DB resides. And there the content-server then publishes all books to the outside so I can access it while traveling from my ebook-reader.
The other client runs on my PC as this is where I manage new book entries (so I don't need remote access). As you add new books they then are automatically stored on the server where the DB is (just using a network path).
However, same as above with the Linux version: Calibre on the server does not recognize new entries without a restart. A automatic "refresh" reading of the DB would be great so I don't need to restart Calibre anymore.
Has anybody an idea? Or would this be a new feature?

[dlfuller]

Thanks for all the feedback.

I thought it said the calibre-server was being stopped before starting the MAC client. But on re-reading it would appear the OP leaves calibre-server running while updating the library from the MAC, then when the MAC is closed the server is stopped and started. My guess is that this sequence is used to minimise the server unavailability time.

First, I do make a lot of demands on the server. Simply a single user either accessing the server or using a single calibre Mac app on a single Mac. Not both simultaneously. No multiple Mac apps. Minimizing server availability is not a concern. Multiple Android devices do access the server simultaneously.

Yes, the calibre-server remains running, revisions are made to the library, and then the calibre-server is stopped and restarted.

Whether that is an issue depends on how the MAC is mounting the share from the server. It should be mounting it as a networked file system, not a native file system such as HFS+ or an EXT variant.

The calibre library is mounted as a volume on the Mac using SFTP.

So far, this simple approach seems to be working. I have revised and renamed titles without issue. Outside of case sensitivity, I don't know what else to look for, but will still keep an eye out for any glitches.

[twinkybot]

@kovidgoyal any news on "linking" remote libraries?

I also would like to have a Client which can connect to a (headless) server.

[kovidgoyal]

The server web interface can now do all the most important operations -- editing metadata, converting, etc.

As for non-browser client to connect to the server, see https://github.com/kovidgoyal/calibr...e/srv/TODO.rst