Server CPU use

[TypeC]

A while ago, I installed Calibre server on a headless 32-bit Atom (x86) system running Ubuntu 14.04.

Here is some of the output from the Linux "Top" command after 39 days of up time:

Tasks:  75 total,   1 running,  74 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.8 us,  0.3 sy,  0.0 ni, 96.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:    504480 total,   483184 used,    21296 free,    39516 buffers
KiB Swap:   524284 total,    34844 used,   489440 free.   217072 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  980 root      20   0  367520   6568    988 S   5.0  1.3   3637:47 calibre-se+
 8566 squeeze+  20   0  165916 156400   5120 S   1.7 31.0 136:10.34 squeezebox+
19632 mike      20   0    3384   1392   1012 R   0.7  0.3   0:00.13 top
18777 root      20   0       0      0      0 S   0.3  0.0   0:16.61 kworker/0:2
19609 mike      20   0   11168   1844   1076 S   0.3  0.4   0:00.02 sshd
    1 root      20   0    4328   1328    632 S   0.0  0.3   0:04.36 init
    2 root      20   0       0      0      0 S   0.0  0.0   0:01.36 kthreadd

The pertinent bit here is that the CPU time for calibre is substantially bigger than anything else (> 3600s or 60 hours!). This behaviour is constant over restarts.

For comparison, second place goes to a music server (the "squeezebox" entry) that controls 4 hardware music players and transcodes streamed audio on the fly. It's still racked up only about 4% of the time Calibre has used. In contrast, the Calibre library probably hasn't even been accessed once since the last restart.

Is this expected behaviour? What is likely to be causing it? Currently I'm thinking of uninstalling from the server and going back to using it solely on my desktop.

NB The instructions I followed suggested that xvfb should be installed to "in order to sidestep issues caused by running Calibre in a non-display environment". I didn't do this, reasoning that I could add it later if I saw any "issues" - could I be seeing one now?

Thanks for any help anyone can provide.
TypeC

[kovidgoyal]

Umm just because you never accessed your calibre server does not mean that no one else accessed it. Look at the server logs. And you dont need xvfb that advice is obsolete.

[kovidgoyal]

Oh and if you happen to be syncing the calibre library on your server with somewhere else, then everytime the server detects that metadata.db has changed, it has to reload the entire db into memory, which is fairly CPU intensive.

[TypeC]

Thanks for your quick response, Kovid.

Originally Posted by kovidgoyal

Umm just because you never accessed your calibre server does not mean that no one else accessed it.

Hopefully, it does - I'm currently the only one in my house who knows how to connect to it (and it's not exposed to the Internet, or shouldn't be).

It's a static library (the timestamp on Metadata.db indicates it was last modified in October).

I've (now) found "calibre-server.log", which appears also not to have been modified since October 27th (I think this may have been when I installed). There's nothing of note in this file, except some errors which I presumably fixed when I was setting it up.

Are there other logs of interest, or a way to get more verbose output in the one I've found?

[kovidgoyal]

The logs will be in ~/.config/calibre (assuming whatever you used to setup the server did not change the calibre config directory) named

server_access_log.txt
server_error_log.txt

[eschwartz]

Originally Posted by TypeC

A while ago, I installed Calibre server on a headless 32-bit Atom (x86) system running Ubuntu 14.04.

What version of calibre?

[TypeC]

Originally Posted by eschwartz

What version of calibre?

2.41 (sorry meant to look this up initially and forgot). I originally installed in May 2015, so I've seen the same behaviour between whatever version was current then and this one.

Found the proper logs - thanks, Kovid (they were hidden in /root/.config/calibre - presumably because Ubuntu's Upstart runs as root).

Other than at the point of upgrade (with a couple of other entries relating to /cherrypy/lib/reprconf.pyo having been changed) server_error_log.txt contains only a straightforward (and identical) group of entries for each restart:

[27/Nov/2015:12:02:04] ENGINE Bus STARTING
[27/Nov/2015:12:02:04] ENGINE Started monitor thread '_TimeoutMonitor'.
[27/Nov/2015:12:02:04] ENGINE Started monitor thread 'Autoreloader'.
[27/Nov/2015:12:02:04] ENGINE Serving on 0.0.0.0:8004
[27/Nov/2015:12:02:05] ENGINE Bus STARTED

and server_access_log.txt has nothing in it between times I know I've accessed the system myself. For example, here are 4 contiguos lines:

192.168.40.134 - - [27/Oct/2015:17:25:51] "GET /get/thumb_90_120/1377 HTTP/1.1" 200 3259 "http://joggler:8004/browse/category/newest" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:41.0) Gecko/20100101 Firefox/41.0"
192.168.40.134 - - [27/Oct/2015:17:25:51] "GET /get/thumb_90_120/1375 HTTP/1.1" 200 2714 "http://joggler:8004/browse/category/newest" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:41.0) Gecko/20100101 Firefox/41.0"
192.168.40.190 - - [05/Jan/2016:12:30:16] "GET / HTTP/1.1" 200 6109 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:43.0) Gecko/20100101 Firefox/43.0"
192.168.40.190 - - [05/Jan/2016:12:30:16] "GET /static/browse/browse.css HTTP/1.1" 200 7646 "http://joggler:8004/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:43.0) Gecko/20100101 Firefox/43.0"

The whole file looks just like (what I'm sure it is) a log of web requests: loads of "GET" and a few "POST".

[kovidgoyal]

First, dont run the server as root. It should be run as the user that has ownership of the library files.

As for your extra CPU usage, normally, I'd be willing to help you track it down, but I am currently in the middle of rewriting the entire server, so the effort would be largely wasted as the new server has no code in common with the old one. So I suggest you just live with the high CPU usage for a few months until calibre 3.0 is released.

[TypeC]

Originally Posted by kovidgoyal

First, dont run the server as root. It should be run as the user that has ownership of the library files.

I understand the reasons for running with as few permissions as possible. On Ubuntu though, Upstart is the recommended way of starting tasks with the system and (at least on my system) runs as root. It doesn't appear to have a non-hacky way of changing the user for a specific job, although I think later versions (later than the one on Ubuntu 14.04) have introduced a method to acheive this. Almost all the services on my server are running as root and almost all of the packages were downloaded and initialised using scripts from the Ubuntu official repo. It is a custom distro though (for my device) and may not have been built to normal standards.

Other than the fact it is a server (which can potentially interact with the outside world), do you have a specific reason for giving this warning?

Originally Posted by kovidgoyal

As for your extra CPU usage, normally, I'd be willing to help you track it down, but I am currently in the middle of rewriting the entire server, so the effort would be largely wasted as the new server has no code in common with the old one.

Under the circumstances, that's quite understandable and perfectly fine, of course. I'm very grateful to you for providing this wonderful software at all.

[TypeC]

I think I may have fixed this. The blog post I linked to earlier which I followed to get things up and running included "--auto-reload" on the command-line arguments for calibre-server.

If I understand the Calibre docs, there's not a lot of point in that. I removed it and now Calibre is sitting at about 0.3% CPU instead of ~5.5%.

I'll look out in a couple of days, but hopefully that will be it!

Thanks again for the help.

[kovidgoyal]

Yeah auto reload is a definite no-no in production. It is meant for development.

Running internet facing software as root is pretty dangerous, but other than that, calibre does not care.

[TypeC]

Sorry to continue this in a slightly off-topic manner, but it turns out it is relatively straightforward to run Upstart jobs as non-privileged users (even though it appears to be uncommon), even if you are using an older version lacking the setuid / setgid settings.

The script has an "exec" statement for the daemon executable: I just put "sudo -u <user>" between "exec" and the name of the executable and the process owner is now showing as "<user>". I don't know whether this has any downsides, but everything I can think of trying appears to be working correctly.

Hopefully this will help anyone else finding this thread in a similar position to me.

(Aside - I've tried three times to correct the original tutorial post - so fewer people *will* be in the same position, but my comments keep being rejected as spam.)

[eschwartz]

Wow...

I think I prefer Wikis. Too many tutorials are barely good enough to give you inspiration for formulating your own solution.