How to set up a content server on a VPS?

[binarygoddess]

Moving a discussion to the appropriate (I hope!) area..

I am trying to set up a calibre content server on a VPN hosted by linode, so far I've just been learning how to use the tools and securing the server. I've installed apache so I can move my websites over, but I haven't taken any steps toward actually setting up the Calibre content server. Chaley has thoughtfully offered to provide some tips and startup scripts etc. if I would start the thread


This is what he's said so far

If/when you install the calibre content server, be sure to install the V2 beta instead of the V1 release. It has several advantages. 1) It doesn't require you to install a fake X server (xvfb). 2) My experience is that it is faster. and 3) my experience is that it requires less memory.

I can give you my config files and startup scripts if they would be useful. I run calibre behind apache (SSL) in reverse-proxy mode so that apache does authentication. I also use monit to monitor the calibre db and restart the content server when dropbox changes the content.

[chaley]

There are several things to think about. I am assuming that you know at least a bit about linux systems admin. If not, what I say below will make little sense.

The first question is whether or not to use standard ports and to use SSL. The calibre server can be "front-facing" if you wish, but then you are limited to a single username/password. If you want to use port 80 or port 443 (SSL), or if you want more than one username/password, then you probably need to use a front-end web server to deal with these issues. I use apache2. Calibre is running in reverse proxy mode, which means that apache2 fields all the requests and send them to calibre. More later....

Second is authentication. Who can access your library, and how? For me, it is members of my family and 2 or three friends who I want to allow to see my lbrary. I decided that the best solution was to offer individual passwords. To do this, I use apache digest authentication. I chose digest because it is reasonably safe over non-encrypted connections.

Third is ensuring that calibre is serving the latest content. I use dropbox to copy content from my "editing" computer to the server. Dropbox on linux apparently uses a "delete then create" scheme to update files. Unfortunately that means that calibre will never see the changes because it holds open the file that was deleted. To get around this I use "monit" to watch the timestamp of the calibre DB and restart the content server if the DB changes. My calibre restart script copies the DB from dropbox to /tmp and sets the environment variable CALIBRE_OVERRIDE_DATABASE_PATH to point at the copy in /tmp. If the timestamp changes then monit will stop and restart calibre-server.

EDIT: Calibre V2 correctly detects that the db has changed, so the copy and monit test are no longer necessary. That said, I am more comfortable ensuring that the DB does not change underneath a running calibre instance and I have some concern about dropbox conflicts because the db is open RW, so I still use monit to stop and start the calibre server

My apache, monit and calibre-server scripts are attached. Note that the apache script will work with a few changes in the non-SSL channel.

[eschwartz]

Question: I know Dropbox sometimes does funny things to the calibre library if run together -- even on linux, potentially (which doesn't lock files). It is usually recommended to run one then the other.

For instance, I have a calibre wrapper, which rsyncs my libraries to the dropbox folder on closing, (as well as wrapping in calibre-debug for automatic logging even via the .desktop).

In what manner would you suggest replacing that with rsycnc? Would that render monit on the server unnecessary? Would it not really cause any difference, thus making it pointless to replace one with the other?

[kovidgoyal]

@chaley: Is restarting the server via monit really necessary? IIRC, any endpoint in the server uses check_if_modified() which checks the modified timestamp on metadata.db and if it is modified, it automatically closes and re-opens the connection to metadata.db

Since the check uses mtimes on the metadata.db path, it should not matter what technique the sync service uses to update the file.

EDIT: Although I have to say that if dropbox is doing a delete and then create, which is not atomic, as opposed to copy-sync-rename, which is, then re-opening the database connection with a partially synced metadata.db would cause bad things to happen.

[chaley]

It has been a while since I looked at this. I have had this running for almost 18 months.

When I first set up the calibre server, I had the problem that it never reloaded a changed database. I (and monit) did see the timestamp change, so the file was indeed being modified. I surmised without any proof that dropbox was using delete/add instead of open/rewrite, which could account for the problem by leaving calibre with a disconnected inode.

As for atomicity, I have no evidence that dropbox is deleting *before* adding. If I were them I would do the job with a temp file and by messing with hard links. The dropbox linux client I have is in python (all .so files) and includes librsync.so, so I suspect that it is using rsync-like semantics.

Of course, the above doesn't deal the problem of why calibre never reacted to the mtime change. I just looked at the content server code and I can't find where it reopens the db. The db is initiallly opened in server.main.py on line 121. I see places that reread the db but nowhere that actually reopens the db, which it clearly must do if the sync client does its job with links instead of overwriting the file itself. Am I being blind?

The problems using dropbox and calibre together come from dropbox being a 2-way mirroring system. By copying the db there is no possibility of the database's modtime changing, which might happen because calibre opens the db R/W. As a result, dropbox will never want to copy it up to the cloud, eliminating the possibility of conflicts. Using a one-way sync system eliminates this concern.

In the end, using monit, copying the db, and telling calibre to use that copy does three things for me. The first is that it works. The second is that it addresses a bit of my paranoia by ensuring that the server is cleanly started using a db that won't be tampered with. The third is another bit of paranoia: that there is something watching in case the server crashes for some reason. I use the same technique for other daemons such as sendmail (and its milters) and dropbox.

[kovidgoyal]

grep the code for reopen() and check_if_modified()

Edit: Line 53 of server/utils.py and Line 126 of db/legacy.py

[chaley]

Originally Posted by kovidgoyal

grep the code for reopen() and check_if_modified()

Edit: Line 53 of server/utils.py and Line 126 of db/legacy.py

Thanks. Tracing that back through expose(), the server dispatch controller, and endpoint management was fun.

I note that the old DB didn't do the reopen. Instead db2.check_if_modified() did a cache refresh, which is the behavior I was seeing when I first set this up (calibre v0.9 releases). The new DB does it right.

If I was not copying the DB then I could let calibre do the timestamp check instead of monit. However, as I want to keep copying the DB to eliminate any chance that dropbox will be "conflicted", monit has to stay.

[kovidgoyal]

Originally Posted by chaley

I note that the old DB didn't do the reopen. Instead db2.check_if_modified() did a cache refresh, which is the behavior I was seeing when I first set this up (calibre v0.9 releases). The new DB does it right.

Indeed, this was one of the things I remember making it a particular point to fix, which was why I was surprised at the need to restart the server.

[binarygoddess]

Chaley: I have more than a passing familiarity with linux administration I was a network admin in a mixed environment for quite a while, even used to run my own webserver... but to be fair it was maybe longer ago than I care to admit since the last time I got my hands dirty. It's been interesting getting up to date again, kind of fun

I'm still migrating our websites, mostly because doing so has brought to my attention how much work has needed to be done on them. They've been languishing too long, poor things! Not that they are all that important in the grand scheme of my life, but I've had the domain over... 12 years? I am loathe to let it go... hehe.

So I am watching this thread with great interest!

[binarygoddess]

Haha! I am happy, everything is in place and working the way I want it to! Thanks Chaley, so much, for the help, the config files were VERY useful

[chaley]

Originally Posted by binarygoddess

Haha! I am happy, everything is in place and working the way I want it to! Thanks Chaley, so much, for the help, the config files were VERY useful

Excellent!

You are using a Linode basic package: 1 GB ram etc? And apache2/mysql/php etc for your web server? And finally, are you seeing periods where the VPS swaps? When the base Linode package was 512MB ram I had lots of performance problems and had to go up one level. I downgraded when they doubled the ram on all the plans.

[binarygoddess]

I do have the basic package with apache2/mysql/php, and I haven't noticed any interruptions so far, but it's only been a few days so I will keep my eye on it. Honestly this is so cool, and there's so much more potential there than for normal shared web hosting, we might end up going up to the next level anyway!