Calibre In Process Server help

[DouglasK]

I'm seeing a 500 error from Apache trying to load it. I've had the in process server running before without issue (on Ubuntu Linux), but have recently had to rebuild the server due to a h/w failure. Version of Ubuntu is now 13.10, This updated the Apache server from 2.2 to 2.4, not sure if that's relevant.

Any help is appreciated.

In short, Calibre is saying it can't find calibre-wsgi-adapter.py. Here's (what I hope are) the relevant logs and configs:

Here's the error from the Apache logs:

[Mon Mar 03 10:45:27.971954 2014] [:error] [pid 6767] [client xx.xx.204.206:52282] mod_wsgi (pid=6767): Target WSGI script '/var/www/cgi-bin/calibre-wsgi-adapter.py' cannot be loaded as Python module.
[Mon Mar 03 10:45:27.971987 2014] [:error] [pid 6767] [client xx.xx.204.206:52282] mod_wsgi (pid=6767): Exception occurred processing WSGI script '/var/www/cgi-bin/calibre-wsgi-adapter.py'.
[Mon Mar 03 10:45:27.972012 2014] [:error] [pid 6767] [client xx.xx.204.206:52282] Traceback (most recent call last):
[Mon Mar 03 10:45:27.972058 2014] [:error] [pid 6767] [client xx.xx.204.206:52282]   File "/var/www/cgi-bin/calibre-wsgi-adapter.py", line 31, in <module>
[Mon Mar 03 10:45:27.972119 2014] [:error] [pid 6767] [client xx.xx.204.206:52282]     from calibre.library.server.main import create_wsgi_app
[Mon Mar 03 10:45:27.972142 2014] [:error] [pid 6767] [client xx.xx.204.206:52282] ImportError: No module named calibre.library.server.main

Here's the output of calibre-debug --paths:

$ calibre-debug --paths
CALIBRE_RESOURCES_PATH=/opt/calibre/resources
CALIBRE_EXTENSIONS_PATH=/opt/calibre/lib/python2.7/site-packages/calibre/plugins
CALIBRE_PYTHON_PATH=/opt/calibre/lib/python2.7:/opt/calibre/lib/python2.7/plat-linux2:/opt/calibre/lib/python2.7/lib-dynload:/opt/calibre/lib/python2.7/site-packages

Here's the content of calibre-wsgi-adapter.py:

# WSGI script file to run calibre content server as a WSGI app

import sys, os


# You can get the paths referenced here by running
# calibre-debug --paths
# on your server

# The first entry from CALIBRE_PYTHON_PATH
sys.path.insert(0, '/opt/calibre/lib/python2.7')

# CALIBRE_RESOURCES_PATH
sys.resources_location = '/opt/calibre/resources'

# CALIBRE_EXTENSIONS_PATH
sys.extensions_location = '/opt/calibre/lib/python2.7/site-packages/calibre/plugins'

# Path to directory containing calibre executables
sys.executables_location = '/opt/calibre'

# Path to a directory for which the server has read/write permissions
# calibre config will be stored here
os.environ['CALIBRE_CONFIG_DIRECTORY'] = '/var/www/books/calibre-config'

del sys
del os

from calibre.library.server.main import create_wsgi_app
application = create_wsgi_app(
        # The mount point of this WSGI application (i.e. the first argument to
        # the WSGIScriptAlias directive). Set to empty string is mounted at /
        prefix='/calibre',

        # Path to the calibre library to be served
        # The server process must have write permission for all files/dirs
        # in this directory or BAD things will happen
        path_to_library='/home/douglask/CalibreWeb',

        # The virtual library (restriction) to be used when serving this
        # library.
        virtual_library=None
)

del create_wsgi_app

[kovidgoyal]

You cannot use the in process server with a binary calibre install, as stated in the User Manual. Use the reverse proxy.

[DouglasK]

Originally Posted by kovidgoyal

You cannot use the in process server with a binary calibre install, as stated in the User Manual. Use the reverse proxy.

Fair enough ... per the docs it should work with a current version installed by the package manager. I include the link to the docs page to ensure we're both referring to the same document.

I initially tried the version through the package manager, and have now reverted to that. The installed version is 1.22.0 .

Now what I'm seeing is the browser sitting on "Connecting". In the server error log (log level is set to debug) I'm seeing the following:

[Wed Mar 05 12:16:11.147060 2014] [authz_core:debug] [pid 13378] mod_authz_core.c(802): [client 66.46.204.206:44211] AH01626: authorization result of Require all granted: granted
[Wed Mar 05 12:16:11.147166 2014] [authz_core:debug] [pid 13378] mod_authz_core.c(802): [client 66.46.204.206:44211] AH01626: authorization result of <RequireAny>: granted
[Wed Mar 05 12:16:11.147535 2014] [authz_core:debug] [pid 13378] mod_authz_core.c(802): [client 66.46.204.206:44211] AH01626: authorization result of Require all granted: granted
[Wed Mar 05 12:16:11.147624 2014] [authz_core:debug] [pid 13378] mod_authz_core.c(802): [client 66.46.204.206:44211] AH01626: authorization result of <RequireAny>: granted
[Wed Mar 05 12:16:11.159324 2014] [:info] [pid 13378] mod_wsgi (pid=13378): Create interpreter 'fog.org|/calibre'.
[Wed Mar 05 12:16:11.160495 2014] [:info] [pid 13378] [client 66.46.204.206:44211] mod_wsgi (pid=13378, process='', application='fog.org|/calibre'): Loading WSGI script '/var/www/cgi-bin/calibre-wsgi-adapter.py'.

Any guidance for getting the in process server working is appreciated.

[kovidgoyal]

I'm afraid I dont have the time to help, just use the reverse proxy. There is absolutely no advantage to using an in process server.

[DoctorOhh]

Originally Posted by DouglasK

Fair enough ... per the docs it should work with a current version installed by the package manager. I include the link to the docs page to ensure we're both referring to the same document.
...~~~...
Any guidance for getting the in process server working is appreciated.

Since you linked to the docs you must have seen this note.

Note: For this to work, all the dependencies needed by calibre must be installed on your system. Doing so is highly non-trivial and you are encouraged not to use in process servers. You will not get any assistance with debugging in process server problems.

[eschwartz]

Originally Posted by DoctorOhh

Since you linked to the docs you must have seen this note.

Kovid just changed that. I'm pretty sure it was in response to this thread.

See: https://github.com/kovidgoyal/calibr...dbd346ba192890

[DouglasK]

Originally Posted by eschwartz

Kovid just changed that. I'm pretty sure it was in response to this thread.

See: https://github.com/kovidgoyal/calibr...dbd346ba192890

Yup, it's a new addition to the page.

I'm suspecting that it has to do with the move from Apache 2.2 to Apache 2.4. The In Process server used to work just fine for me using the same distro packages. I've seen other mentions of folk having issues after moving to Apache 2.4 as well.

Here's why I'm not big on running the Calibre server and doing a reverse proxy: It keeps several processes running, each gobbling 115Mb (approx) of resident memory, and 3.3GByte of virtual memory. In my opinion, that's a heavy load to continually run on a server even when it's not being accessed. iirc, the in process server was a tad slower, but would load only on an as needed basis. To me, this is far preferable in a scenario where I'll access it primarily when I'm done a book and looking for something to read.

As for Kovaid, what, if any support he gives is 100% up to him. I expect he has a life outside Calibre and hope he puts that first!

I'd love to see development time put into the in process server, but I realize it's not a feature that most people use, and as such should be prioritized down, especially with things like COPS in existence.

I may look into how to get COPS to play nice with the web browser in my Kobo Aura HD. (Currently, that browser will not download from COPS when the download button is clicked, but downloading the same file / format from the Calibre server works fine)

[DoctorOhh]

My apologies for coming to the party late.

I'm glad Kovid clarified his position.

[kovidgoyal]

@DouglasK: Dont be fooled by top. The resource requirements for the in process server and the reverse proxy are identical. top is not an accurate way to determine a processes resource requirements. Just try the simple expedient of summing up the memory usage numbers of all processes reported by top and see that you will get many multiples of the actual physical resources on your machine.

A bit of googling should get you more info on that.

[kovidgoyal]

In fact, depending on the MPM you use in Apache, resource consumption could be several times higher. Since if you chose the process MPM you will get one copy of the calibre server using its own resources per worker process instead of just a single process with threads when using the reverse proxy.

[Ekkehard.Blanz]

Kovid, I can easily see your point. However, you are describing a different use case, i.e. lots of and frequent requests. @DouglasK and I have a different use case: very rare requests, almost guaranteed not to be simultaneous, where it would be advantageous if the calibre server would not be loaded all the time. Therefore, I think your initial architectural decision to provide both must be applauded. I also understand that you are hesitant to support the inProcess approach - it requires the distro (not binary) version, and those are notoriously behind the latest revision. At any rate, apache 2.4 broke both approaches the way they are described in the docs. While I had no luck with the inProcess version, I could get the reverse Proxy to work. Here is how (YMMV):
LoadModule proxy_module /usr/lib/apache2/modules/mod_proxy.so
LoadModule proxy_http_module /usr/lib/apache2/modules/mod_proxy_http.so

ProxyPass /calibre http://localhost:8080/calibre
ProxyPassReverse /calibre http://localhost:8080/calibre

SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1

[kovidgoyal]

If you want to cater to a use case where the calibre server is only needed very rarely, put it behind a proxy that starts it on demand and shuts it down after a period of inactivity.

[kovidgoyal]

Actually, I dont see the point of an on-demand server. A running calibre server uses no resources and should get swapped out if no requests arrive for a while and the computer is under memory pressure. So shutting it down is pretty pointless. Not to mention that I doubt Apache shuts itself down when there is no demand either. So the resource consumption would be no different in either scenario.