Multiple Libraries from a single server?

[Mex5150]

Hi All

I have 12 libraries (all pertaining to different topics, so it makes more sense to keep them like that rather than all on one huge library). I've just started playing about with the Calibre Server, and the only drawback is it seems to only be able to serve on library at a time. A quick google says this is indeed all it can do, but all of these articles seem to be quite old, and knowing the insane rate Calibre updates come out, they may no longer be accurate. Can it do more now? And if so how?

I know another way to do this is to just run twelve instances of the CLI server, which frankly seems a very inelegant way of doing things, but also raises another question. Can the GUI version and the CLI version of Calibre coexist? i.e. if I have the CLI version serving a library, but then use the GUI version to edit a book in the same library will things break, or will it update without a hitch?


~Mex

[theducks]

Originally Posted by Mex5150

Hi All

I have 12 libraries (all pertaining to different topics, so it makes more sense to keep them like that rather than all on one huge library). I've just started playing about with the Calibre Server, and the only drawback is it seems to only be able to serve on library at a time. A quick google says this is indeed all it can do, but all of these articles seem to be quite old, and knowing the insane rate Calibre updates come out, they may no longer be accurate. Can it do more now? And if so how?

I know another way to do this is to just run twelve instances of the CLI server, which frankly seems a very inelegant way of doing things, but also raises another question. Can the GUI version and the CLI version of Calibre coexist? i.e. if I have the CLI version serving a library, but then use the GUI version to edit a book in the same library will things break, or will it update without a hitch?


~Mex

Just start each of them with a separate port and Libraryname

"C:\Program Files\Calibre2\calibre-server.exe" --port 8083 --with-library C:\Users\yyyy\MainLib

IMHO Less Libraries is better. A Virtual Library may accomplish much of what you need.

Reasons to keep separate:
Structure (columns or allowed values) differences
Content filter by audience (PG13,...)
non-Shareable content (vs public domain)

[Mex5150]

Originally Posted by theducks

Just start each of them with a separate port and Libraryname

"C:\Program Files\Calibre2\calibre-server.exe" --port 8083 --with-library C:\Users\yyyy\MainLib

I know I can do this, but as I said it's a very inelegant way of doing it. It will waste a lot of resources, and it also means jumping port every time I want to switch libraries.

IMHO Less Libraries is better. A Virtual Library may accomplish much of what you need.

Reasons to keep separate:
Structure (columns or allowed values) differences
Content filter by audience (PG13,...)
non-Shareable content (vs public domain)

I've been using Calibre for many years now and have built up these libraries (most of which do indeed have different structures), I really don't want to dump them all in one mega-library just to do one simple thing.

Assuming I am forced to go with the multiple cli instances, as I asked above how well do the cli and gui versions play together?

Oh, I'm running on Linux by the way, if that affects anything.


~Mex

[theducks]

They play nice

Kovid made them do it




The old saw... Just because you did it that way years ago...


Calibre did not have VL's years ago. You had no choices then.

And yes, I would expect a large, single would use less resources than multiple.

So you have another choice (or at least, a test case) to make

[eschwartz]

The CLI server doesn't really use a lot of resources...
Let me check my servers (I have systemd user services running for them)...
Well. Currently running at a combined 0.5% CPU and 3% memory.



I imagine it may start to take up more resources once I actually hit them up.



As for editing in the GUI, the server will automatically pick up the new changes.

[eschwartz]

And do note that the currently-underway server rewrite handles multiple libraries in one process, and I can only assume from the GUI as well.

Keep an eye out for calibre v3.0, coming... sometime vaguely soon-ish. (a/k/a whenever it is ready.)

[Mex5150]

Originally Posted by theducks

They play nice

Kovid made them do it

That's great news, I was rather concerned (I remember having problems with two instances of the GUI accessing the same library on a network drive in an early experiment years ago).

The old saw... Just because you did it that way years ago...


Calibre did not have VL's years ago. You had no choices then.

And yes, I would expect a large, single would use less resources than multiple.

So you have another choice (or at least, a test case) to make

I know it's concidered a cardinal sin in the Calibre community, but multiple libraries makes accessing the files directly outside of Calibre a lot easier too, so no plans to change.

Originally Posted by eschwartz

The CLI server doesn't really use a lot of resources...
Let me check my servers (I have systemd user services running for them)...
Well. Currently running at a combined 0.5% CPU and 3% memory.

Cool ;^>

And do note that the currently-underway server rewrite handles multiple libraries in one process, and I can only assume from the GUI as well.

I'm not bothered about running the server from CLI as lang as it plays nice with the GUI version.

Keep an eye out for calibre v3.0, coming... sometime vaguely soon-ish. (a/k/a whenever it is ready.)

I know this is probably a 'how long is a piece of string?' question, but any idea on when that may be? I have nothing really urgent to serve, so waiting a week or two for the new version is fine, but if it's going to be a few months, I'll just get on with the multiple CLI versions in the meantime ;^>


~Mex

[theducks]

2 GUI on same Library is a No No
GUI and server is good

[eschwartz]

Originally Posted by Mex5150

That's great news, I was rather concerned (I remember having problems with two instances of the GUI accessing the same library on a network drive in an early experiment years ago).

The CLI server never writes to the library, so it is always safe.
(Well, if the database schema is upgraded then the CLI server will perform that update...)

I know it's concidered a cardinal sin in the Calibre community, but multiple libraries makes accessing the files directly outside of Calibre a lot easier too, so no plans to change.

Not a cardinal sin, but you should know why you chose to do so, and make up your own mind whether it makes sense to use Virtual Libraries instead.

I make extensive use of both VLs and multiple physical libraries.

I know this is probably a 'how long is a piece of string?' question, but any idea on when that may be? I have nothing really urgent to serve, so waiting a week or two for the new version is fine, but if it's going to be a few months, I'll just get on with the multiple CLI versions in the meantime ;^>


~Mex

Just go with multiple CLI incantations.

When calibre v.30 comes out, you can upgrade your startup routine.
Multiple CLI invocations aren't really a bad thing, honestly... as I said, it's pretty lightweight, so why not?


Also, what distro do you use (I use ArchLinux), and does it use systemd as its init system?
Here is a systemd unit file which starts up a calibre library via drop-in config files.

[eschwartz@arch ~]$ cat ~/.config/systemd/user/calibre-server@.service
[Unit]
Description=Main calibre server
After=network.target

[Service]
EnvironmentFile=/%h/.config/calibre/servers/%i.conf
ExecStart=/usr/bin/calibre-server \
    --port "${port}" \
    --with-library "${path}" \
    --restriction "${restriction}" \
    --user "${user}" \
    --pass "${pass}"

[Install]
WantedBy=default.target

In your calibre configuration directory, create a folder called "servers", add ${library_name}.conf files, and keep the various parameters in the conf file.

[eschwartz@arch ~]$ cat ~/.config/calibre/servers/amazon.conf
port=""
path="/home/eschwartz/Calibre Libraries/Amazon"
restriction=""
user=""
pass=""

Then start a library with

systemctl --user start calibre-server@libraryname

or enable it every time you login with

systemctl --user enable calibre-server@libraryname

...

When will the new server be available? I don't know.
The rewrite started in May-June (basic infrastructure), took a siesta until October (another tool needed was being worked on outside of calibre's codebase), and you can see the current progress using

calibre-debug --new-server

As you can see there is still a ways to go. It's being rewritten from the ground up. I would go with a few months.
Hey, we have a partially working demo.

I just stick with "exciting things are ahead".

[Mex5150]

Originally Posted by theducks

2 GUI on same Library is a No No

Yea, it was just an experiment.

GUI and server is good

;^>

Originally Posted by eschwartz

The CLI server never writes to the library, so it is always safe.

;^>

Not a cardinal sin, but you should know why you chose to do so, and make up your own mind whether it makes sense to use Virtual Libraries instead.

I've been shouted at several times for wanting to do so LOL

Just go with multiple CLI incantations.

I will, I just thought there's no point setting it all up just to have the new version that will do it all in one hit released a day or two later LOL

Also, what distro do you use

I use Arch on one desktop, Ubuntu on the other, and Kali on the laptop.

(I use ArchLinux), and does it use systemd as its init system?
Here is a systemd unit file which starts up a calibre library via drop-in config files.

[eschwartz@arch ~]$ cat ~/.config/systemd/user/calibre-server@.service
[Unit]
Description=Main calibre server
After=network.target

[Service]
EnvironmentFile=/%h/.config/calibre/servers/%i.conf
ExecStart=/usr/bin/calibre-server \
    --port "${port}" \
    --with-library "${path}" \
    --restriction "${restriction}" \
    --user "${user}" \
    --pass "${pass}"

[Install]
WantedBy=default.target

In your calibre configuration directory, create a folder called "servers", add ${library_name}.conf files, and keep the various parameters in the conf file.

[eschwartz@arch ~]$ cat ~/.config/calibre/servers/amazon.conf
port=""
path="/home/eschwartz/Calibre Libraries/Amazon"
restriction=""
user=""
pass=""

Then start a library with

systemctl --user start calibre-server@libraryname

or enable it every time you login with

systemctl --user enable calibre-server@libraryname

That'll come in handy, thanks ;^>

I just stick with "exciting things are ahead".

Indeed, looking forward to it.


~Mex

[eschwartz]

Happy to help.

Some people approve of multiple physical libraries, some disapprove. I take the opinion that it is all good as long as there was an intelligent decision involved.
Not my job to tell someone their thought-out decisions are stupid. It is possible I will scream bloody murder if they make decisions for no reason at all.


One good reason to use multiple physical libraries, for example, is if they use wildly different sets of custom columns.

Originally Posted by Mex5150

Indeed, looking forward to it.

I am looking forward to it as well, but one of my big reasons for doing so is in order to resolve the issue of "stupid distros patch out forked/modified-from-upstream modules".
calibre v3.0 apparently won't use cherrypy, that'll stop people from complaining that their calibre-servers don't work with auth. Eventually they tell us they are using the distro package...
Even Arch isn't immune.