Calibre 3.0 Content Server standalone

[Terisa de morgan]

Originally Posted by Rayha

I don't know too much about the new functionallity in content server, and what information it actually write to the database. But my guess is that it have to do with the reading books functionallity. And even though it is possible to read books through GUI as well, the normal maintenance done in GUI, like updating metadata, adding books and new formats of books, will not conflict with data that is related to reading functionallity.

I'm afraid you're wrong, as kovid has said that he intends to add to content server all the functions for normal maintenance, so yes, both will conflict.

[HendelTBD]

I was about to start a new thread when I found this one, which seems to explain my problem - I too host my Calibre library on a NAS, run the content server on one (headless) Mac, and add/maintain content from a copy of Calibre running on a different Mac. I start and keep the server process running via LingonX - been doing all of this for quite a while.

All of this works fine, except I'd noticed that with 3.x when I added new content to the library, it wasn't showing up in the content server web interface, or the OPDS listings, until I restarted the content server.

Would a command-line option to run the content server as a strictly read-only process solve any of this? I'm trying to schedule a nightly re-start of the content server process (again with LingonX) as a workaround.

[snoopy_1978]

Hi everybody,

just jumping in this thread as I actually upgraded from Calibre 2.x to Calibre 3.1.1 and stumbled across the various architectural changes in the 3.x version.
In the 2.x Version I used multiple calibre-server instances bound to different ports with different url-prefixes and different virtual-libraries which in the new version is quite different / harder / even not possible(?) to implement.

Background and solution in V2:

• I use one big calibre library where all books are in.
• For accessing the library from the internet I use the standalone calibre-server running on a headless linux server (virtual) machine. This should ever be read only!
• I have different groups of persons who should see different sub-sets (=virtual libraries) of the whole content (a simple example: Adult and Non-Adult content)
• An Apache Server exists since many years which listens on Port 443 and acts as a central web portal which serves other services (nextcloud for example) beside and also calibre-server (WebFrontend and OPDS are in use).
• The portal in Apache is completely configured to use Basic Auth with an LDAP Backend for centralized user / group management, so at this point no unauthorized access also (but not only) to calibre is implemented. Therefore it isn't really neccessary to use the authentication mechs of calibre.
• In Apache the different calibre-instances are included by ReverseProxy directives pointing to the different url-prefix/port combinations of the different calibre-server instances.
• The Apache portal now gives access to the different calibre-urls based on the logged in LDAP User.
• The router just forwards port 443 to this Apache Server, no forwarding to the calibre-server instances Ports.

So far so good, this worked as expected.. the different LDAP Users got different virtual libraries in the portal and so - for example - children didn't get access to adult content.

Now I wanted to adopt this behaviour to V3 and came across these issues which I could partially solve:

• No --virtual-libray Option in V3
  Solved by creating the calibre userdb, a few different (technical) users with restricted access to other parts of the library. These users don't have write access.
• Apache LDAP login now conflicted with the now enabled Basic Auth of calibre-server in the backend
  Solved by inserting "RequestHeader set Authorization..." in the responsible Apache directive with the ReverseProxy for calibre

Now I stuck at the point to start multiple calibre-server instances for presenting different url prefixes that map to different calibre-users and so to different virtual libraries.
As mentioned in this thread, it is not possible to start different server instances any more?? When one calibre-server is running and I want to start a second one (with different options but the same big library) I get this error:
"Another calibre program such as another instance of calibre-server or the main calibre program is running. Having multiple programs that can make changes to a calibre library running at the same time is not supported"

The part "that can make changes to a calibre library" is the important one as the calibre-server instances (in my case) never ever will do any changes as they should just give read-only access to the library.
Isn't there any option to tell calibre-server it should run in a somehow "global read-only" mode (as this would be absolutely enough)? In this case, data-loss can't occour with concurrent writes. I just tried "--disable-local-write" but this didn't work, the error remains.

IMHO there could be different solutions to solve this:

• Give an option like "force (I know what I do)" or "global readonly" to the calibre-server so it will be possible to run different instances like in V2
• Get rid of the "url-prefix" and let the server automatically determine the prefix (as many many other web-services do today, nextcloud for example). So there would be no need to start different servers, one would be enough. The different portal URLs could then be proxied to just one single calibre-server in the backend. Access then will be realized by using different "RequestHeader set Authorization" directives in Apache.

Or did I miss something? Or does anyone have another hint how I could achieve the V2 solution in V3?

Greetings and thx in advance!
Snoopy

[kovidgoyal]

@snoopy: A single content server instance can now serve all your libraries. So there is no need to run multiple instances.

@HendelTBD: I'm afraid there is not going to be an option to run the server readonly, this is because the server is now fundamentally read/write in that many core operations it does require write access to the db. As time passes more and more of these operations will be exposed in the main server interface and trying to maintain two different versions of that interface is too difficult. So if you want to simulate readonly behavior, your best bet is to have a script that rsyncs the library to the server computer, and restart the the content server there.

[snoopy_1978]

@kovid
Because of your very common statement which can just be found in the calibre docs, I think you didn't really read / understood my post, do you?
It's absolutly clear that one calibre-server can serve multiple virtual libraries with the help of the new userdb in a very simple use case. I absolutly understood how this works as I implemented that for testing without the apache proxy, no problem.
But with the current restriction (no multiple calibre-server instances allowed) my use case I described above is much harder / impossible(?) to implement.

Please answer this core question resulting of my scenario and the restrictions you implemented with V3:

How can a single (existing) apache with enabled Basic Auth and LDAP Backend as own User / Group Management act as a single-entry-point ReverseProxy for multiple virtual libraries if just one calibre server with one url-prefix and one port may run?

It is NOT possible to have multiple Basic Auths (first the one from apache and second the one from calibre afterwards) one after the other.
To explain this further:
You open the URL of the apache reverseproxy for one calibre-server lets say:

https://myhost.dyndns.org/calibre

(calibre-server in the backend runs with "--url-prefix /calibre" and enabled auth with a userdb)

Now this happens:

• The BasicAuth of Apache kicks in and you have to enter valid credentials of a LDAP User
• Directly afterwards the calibre BasicAuth wants to have valid credentials of a calibre user (mapping a virtual library)
• Every following request (XHR, whatever...) fails with a "401 Unauthorized" by calibre, because the browser always sends the FIRST BasicAuth credentials (from the Apache)

Just try urself... So I ask u again: How would u solve this problem with V3?

(BTW: In general it would be not very comfortable to have to enter different credentials twice for entering a service even it would technically work...)

[kovidgoyal]

Umm just use the Virtual library facility inside the server. i.e. any user that wants to restrict the view to a virtual library can simply choose that virtual library by clicking the three dots in the top right corner and choosing Virtual Library.

Instead of describing your somewhat convoluted existing setup, why dont you describe what you are actually trying to do. And I dont see what apache with LDAP as a reverse proxt has to do with anything. The calibre server does not care how it gets the authentication headers, whether directly from a browser or via a proxy in the middle.

[snoopy_1978]

Originally Posted by kovidgoyal

Umm just use the Virtual library facility inside the server. i.e. any user that wants to restrict the view to a virtual library can simply choose that virtual library by clicking the three dots in the top right corner and choosing Virtual Library.

You mean the V2 - WebFrontend? This is a point, right, but the OPDS interface honors --with-library, so for me this is enough.
What I never tried is if you can restrict access to this button / "choose - virtlib site" with the help of Apache, but for me this is not as important.
And with V3 thanks of the new userdb (in general a good idea I think) access can be restricted.

Originally Posted by kovidgoyal

Instead of describing your somewhat convoluted existing setup, why dont you describe what you are actually trying to do.

What I'm trying to do is to achieve the same (or similar) behaviour that worked in V2 with the architecture changes you made with V3. So I HAVE to descibre what I have now with V2 as this is exactly what I'm trying to do in V3.
Perhaps some pictures will clear things up. In the attachments in "calibre-V2.png" you find the architecure I implemented with calibre-server V2.
As you can see, I have with V2:

• An Apache vHost with many Locations (not just calibre), each using BasicAuth ldap

Concerning calibre:

• 2 different Locations:
  • /calibre, which ends up in a (lets say) "NonAdult" virt.lib.
  • /calibre-all, which ends up in the whole lib.
• This could be realized because calibre-server can be startet multiple times with different "--url-prefix" / "--port" and "--with-lib".
• To the Apache Location "/calibre-all" just some users have access (lets say "John" and "Jane", but NOT "Sandra").
• To the Apache Location "/calibre" (which serves the virt.lib "NonAdult") the users "John", "Jane" and additionally "Sandra" have access.

Now I want the same in V3, nothing more, nothing less.

For the following descriptions look in the attachments at "calibre-V3.png":
As you can see here, just one calibre-server can be started, as u restricted that and thefore the "--virt-lib" option doesn't make sense of course.
What is more: because of the need of "--url-prefix" it isn't possible any longer to have different Apache-Locations ReverseProxying now the only calibre-server as the "Location" directive in Apache has to match the "--url-prefix" Option in calibre, otherwise calibre builds the wrong URLs.
And so I can just have one Apache Location "/calibre" to which all LDAP Users ("John", "Jane" and "Sandra") now have to have access. With V2 this was the point where I could control / restrict access.

But how can access now be restricted for "Sandra" to the "NonAdult" Virt.Lib like before in V2?
In V3 this can just be achieved with enabling the calibre-auth and - in the new userdb - mapping users to virt.libs.
In my example u can see two (technical) users in the calibre-userdb. "All", which is not restriced, and "NonAdult" which is restricted to the virt.lib. "NonAdult". (To be exact: a SearchString filtering out Books with concerning tags)

BUT: This implementation does not work at all because of the two BasiceAuths (Apache + calibre) following each other, which leads me to your next statement:

Originally Posted by kovidgoyal

And I dont see what apache with LDAP as a reverse proxt has to do with anything. The calibre server does not care how it gets the authentication headers, whether directly from a browser or via a proxy in the middle.

The point is not, HOW calibre gets the AuthHeaders from the browser but WHAT a browser sends in a request. As I said, try yourself (a simple FileBased Auth with a .htusers File will be enought, it doesn't have to be a ldap backend of course).
Will try again to describe in more detail, why this can't work (look at image "calibre-V3.png"):

1. Browser requests https://myhost.dyndns.org/calibre
2. Apache wants to have valid LDAP credentials (first basic auth)
3. User enters valid LDAP credentials (User: "John"), Apache accepts them
4. ApacheProxy forwards the request to calibre
5. Calibre wants to have valid calibre-credentials (second basic auth)
6. User enters calibre credentials (User: "All") (BTW:ugly, entering different credentials two times)
7. Calibre accepts the credentials
8. CalibreWebFrontend in the Browser requests more content (images, js, css, XHR...)
9. Now the important point: Therefore the Browser has again to send AuthHeaders with each following request. But which one? The first (from the ApacheAuth)? Or the second (from the CalibreAuth)? The browser decides the first, the Apache Ones (User: "John")
10. Calibre gets the AuthHeader with the Apache Credentials (User: "John", NOT User "All")
11. Calibre send a 401 - Access denied for each following request.

So again my question:
How would u build this scenario in V3 (Basic auth enabled Apache as Proxy and calibre with restricted virtual - libs)?

[kovidgoyal]

I dont understand why you are having a problem. You want to restrict users to particular libraries/virtual libraries, use the userdb feature for that. And let calibre do the authentication instead of apache, just use a script in crontab to update the calibre db with the users from ldap. The calibre userdb is a simple sqlite db.

[kovidgoyal]

In fact, in the next release I will modify the --manage-users option to calibre-server so that you can use it to easily programmatically manipulate the user database, if needed.

[snoopy_1978]

Originally Posted by kovidgoyal

I dont understand why you are having a problem. You want to restrict users to particular libraries/virtual libraries, use the userdb feature for that. And let calibre do the authentication instead of apache, just use a script in crontab to update the calibre db with the users from ldap. The calibre userdb is a simple sqlite db.

May be a way... But how would I update the passwords? In the LDAP the passwords are hashed (of course, so are they in the calibre db, I hope, everything other will be a security risk) and of course I don't know the passwords as users manage them by theirselfes.

[kovidgoyal]

You have hashed passwords then you have to work a little more. Have your script insert some default password into the calibre database and have apache re-write the Authorization header to use that default password instead (also make sure you pass --auth-mode=basic to the calibre server).

[kovidgoyal]

Oh and since the calibre server supports digest authentication with the MD5-SESS algorithm, not just basic, it cannot store hashed passwords.

[snoopy_1978]

Another option I'm currently playing around witht is the "RequestHeader" directive in Apache.
With this it is possible to send / modify the AuhtorizatonHeader to the backend server (calibre in this case).
Technically this works, in the location "/calibre" I inserted

RequestHeader set Authorization "Basic <base64 encoded User:Pass String>"

And now after entering the correct LDAP credentials, calibre doesn't ask for a user, as apache send correct AuthHeaders to calibre now.

But now I had to decide somehow, which User:Pass String I have to send from Apache when user "Sandra" logs in or user "John".

Here again different Locations in Apache could help:

• In Location "/calibre" one could use "RequestHeader" with the auth string of calibre User "NonAdult"
• In Location "/calibre-all" one could use "RequestHeader" with auth string of calibre User "All"

But therefore, the "--url-prefix" option in calibre had to be removed and calibre had to determine the prefix automatically.

[snoopy_1978]

Originally Posted by kovidgoyal

Oh and since the calibre server supports digest authentication with the MD5-SESS algorithm, not just basic, it cannot store hashed passwords.

In this case (even if I would know them) I would NEVER EVER sync productive User/Passwords (which are used as Single-Sign-On in other Services) from an LDAP to the calibre UserDB. If this sqlite database gets stolen somehow... don't want to think further...

[kovidgoyal]

Originally Posted by snoopy_1978

In this case (even if I would know them) I would NEVER EVER sync productive User/Passwords (which are used as Single-Sign-On in other Services) from an LDAP to the calibre UserDB. If this sqlite database gets stolen somehow... don't want to think further...

You are syncing default passwords, sow hat earthly difference does it make?