Calibre Content Server Not Updating

[josh.p.23]

So, I have this setup

1) Calibre running on my Ubuntu laptop, I add all of my books here, and will typically sync with a device here, if I am going to do this at all, usually, not so much that.

2) Calibre-server running on an Ubuntu server out on the interwebs integrated into apache via reverse proxy

3) I make changes to the library on the laptop, and then update the calibre server databse and file structure using a sync tool called Unison.

4) I reload the calibre server using an init script, reload apache for good measure, and then go to my web browser or even try FB Reader's remote library function to check for the changes and ...

5) Nothing Happens, the content served up is unchanging.

Can anyone help me understand what I am missing here?

[DoctorOhh]

Originally Posted by josh.p.23

3) I make changes to the library on the laptop, and then update the calibre server databse and file structure using a sync tool called Unison.

I think this step may be your problem.

The metadata.db is the face of calibre if this file doesn't get changed then the content server will look as if nothing changed. If calibre on your laptop is still open when you run this it may prevent the sync tool from copying the metadata.db file and thus you don't see a change on your server. Or if the content server is running it has the metadata.db file open and may prevent the sync tool from altering it which again results in the metadata.db file from being changed.

You might find this thread interesting. Even though the thread starts out talking about dropbox the result was a sync issue due to an open database.

Good Luck.

[chaley]

Originally Posted by josh.p.23

3) I make changes to the library on the laptop, and then update the calibre server databse and file structure using a sync tool called Unison.

4) I reload the calibre server using an init script, reload apache for good measure, and then go to my web browser or even try FB Reader's remote library function to check for the changes and ...

There isn't any way that calibre can open a database containing info X and serve info ~X. If it appears to be doing that, then one or more of three things must be true.

1) You are not updating the database that the content server is really using. Check the various environment variables to be sure you haven't directed calibre to get the database somewhere else.
2) You are not actually restarting calibre, perhaps because of some fault in the init script.
3) You have a proxy server between your clients and calibre.

[josh.p.23]

Ok, I made certain to "killall calibre-server" on my server before initiating Unison sync, then PRESTO! It worked! That thread was interesting, thank youso much!

So, is there a preferred way to automate this sort of thing so that I do not have to manually ssh in, stop the server, sync, and restart the server?

I have been looking, but haven't been able to recognize anything.

[chaley]

You can do what I said I do in post #5 of that thread:
- use "monit" to monitor the timestamp of metadata.db. It restarts calibre if the timestamp changes
- use monit to ensure that the server is actually running
- have the calibre start/stop script copy the database to /tmp
- set the environment variable CALIBRE_OVERRIDE_DATABASE_PATH to tell calibre where the db is located

I don't use a start/stop script in /etc/init.d. I let monit take care of it.

I run debian wheezy. The relevant monit sections are

check process calibre-server with pidfile /var/run/calibre-server.pid
  start program = "/root/mon/calibre-server.sh start"
  stop program  = "/root/mon/calibre-server.sh stop"
  alert YOUR_EMAIL_ADDRESS_HERE only on { timeout, nonexist }

check file metadata.db with path /home/dropbox/Dropbox/Charles/Library/metadata.db
  if changed timestamp
     then exec "/root/mon/calibre-server.sh restart"

The start/stop script that monit uses (/root/mon/calibre-server.sh) follows. I blatantly stole it from an init.d script.

Spoiler:

#! /bin/sh
# Author: YOUR NAME HERE
#
# Please remove the "Author" lines above and replace them
# with your own name if you copy and modify this script.
# Do NOT "set -e"
# PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC="Calibre Content Server"
CONTENT=PATH TO YOUR LIBRARY (FOLDER WITH BOOKS) HERE
NAME=calibre-server
DAEMON=/usr/bin/$NAME
PIDFILE=/var/run/$NAME.pid
DAEMON_ARGS="--with-library $CONTENT --pidfile=$PIDFILE --daemonize -p YOUR_PORT_HERE "
SCRIPTNAME=/etc/init.d/$NAME
# Exit if the package is not installed
[ -x "$DAEMON" ] || exit 0
# Read configuration variable file if it is present
[ -r /etc/default/$NAME ] && . /etc/default/$NAME
# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh
# Define LSB log_* functions.
# Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
. /lib/lsb/init-functions

export CALIBRE_OVERRIDE_DATABASE_PATH=/tmp/metadata.db
#
# Function that starts the daemon/service
#
do_start()
{
        # Return
        #   0 if daemon has been started
        #   1 if daemon was already running
        #   2 if daemon could not be started
        cp $CONTENT/metadata.db $CALIBRE_OVERRIDE_DATABASE_PATH
        start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --test > /dev/null || return 1
        start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON -- $DAEMON_ARGS || return 2
}
#
# Function that stops the daemon/service
#
do_stop()
{
        # Return
        #   0 if daemon has been stopped
        #   1 if daemon was already stopped
        #   2 if daemon could not be stopped
        #   other if a failure occurred
 
        start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE 
        RETVAL="$?"
        [ "$RETVAL" = 2 ] && return 2
        # Wait for children to finish too if this is a daemon that forks
        # and if the daemon is only ever run from this initscript.
        # If the above conditions are not satisfied then add some other code
        # that waits for the process to drop all resources that could be
        # needed by services started subsequently.  A last resort is to
        # sleep for some time.
        start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON
        [ "$?" = 2 ] && return 2
        # Many daemons don't delete their pidfiles when they exit.
        rm -f $PIDFILE
        return "$RETVAL"
}
#
# Function that sends a SIGHUP to the daemon/service
#
case "$1" in
  start)
        [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
        do_start
        case "$?" in
                0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
                2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
        esac
        ;;
  stop)
        [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
        do_stop
        case "$?" in
                0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
                2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
        esac
        ;;
  status)
       status_of_proc "$DAEMON" "$NAME" && exit 0 || exit $?
       ;;
  restart)
        #
        # If the "reload" option is implemented then remove the
        # 'force-reload' alias
        #
        log_daemon_msg "Restarting $DESC" "$NAME"
        do_stop
        case "$?" in
          0|1)
                do_start
                case "$?" in
                        0) log_end_msg 0 ;;
                        1) log_end_msg 1 ;; # Old process is still running
                        *) log_end_msg 1 ;; # Failed to start
                esac
                ;;
          *)
                # Failed to stop
                log_end_msg 1
                ;;
        esac
        ;;
  *)
        #echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2
        echo "Usage: $SCRIPTNAME {start|stop|status|restart}" >&2
        exit 3
        ;;
esac

The rest is up to you.

[josh.p.23]

Just to be sure I got this,

1) when Calibre opens the database to read on the server, even though it is not writing to the database file, it locks the file so that it cannot be written to, making syncing a pain

2) to get around this, you can either
a) manually stop the calibre-server, sync, and restart it or
b) use the method which you outlined above

Using monit is nice and hands off. Ok, so that effectively makes it so that

1) when Calibre-server restarts, it copies the metadata.db to /temp, and reads it from there, leaving the "original" database unopened, and therefore unlocked and writable, syncable.

2) whenever the original database (on the server) is altered (synced up from the remote database on the laptop or home desktop installation of Calibre using dropbox, ubuntu one, rsync, Unison, or whatever), monit detects the timestamp alteration and restarts calibre, copying the new databse to temp, replacing the old there.

I got this Thank you for pointing that out, I did not recognize that in the post.

[chaley]

With the possible exception that sync programs may use delete/add instead of overwrite, your synopsis is correct.

Delete/add might be preferred by the sync folks because it avoid many of the locking problems. In this case calibre would be left holding an orphan file with no name, something it would never detect. I copy the dB away just in case. I am not sure it is always needed.

[kovidgoyal]

@chaley: Actually, in newdb, when the last modified time of metadata.db changes, the content server closes and re-opens the connection to the db (as opposed to just refreshing). This should allow it to work with the delete and replace pardigm that sync programs use, though I haven't tested it myself.

[GRiker]

Originally Posted by kovidgoyal

@chaley: Actually, in newdb, when the last modified time of metadata.db changes, the content server closes and re-opens the connection to the db (as opposed to just refreshing). This should allow it to work with the delete and replace pardigm that sync programs use, though I haven't tested it myself.

I just tried this on OS X, with calibre 1.0.0 running on my laptop, and calibre-server running on a separate OS X server. Library synced by Copy.com.

I added a book, changed metadata in another book, then waited for sync to complete.

I opened a web view of calibre-server from a third location and the new books was listed and the metadata change was visible. It works!

G

[chaley]

Works for me as well: linux server running debian wheezy, calibre 1.0.0, synced from my windows desktop by dropbox, using the in-situ db (not a copy), monit turned off.

On the other hand, with 1.0.0 browsing is much slower. Opening the "latest" page after a refresh takes 20 seconds for 2000 books. Using 0.9.44 it takes slightly more than a seconds. For the moment I will keep the older release and use monit.