Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Devices

Notices

Reply
 
Thread Tools Search this Thread
Old 01-05-2016, 07:22 AM   #1
TypeC
Junior Member
TypeC began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Jan 2016
Location: UK
Device: Nook Glowlight
Server CPU use

A while ago, I installed Calibre server on a headless 32-bit Atom (x86) system running Ubuntu 14.04.

Here is some of the output from the Linux "Top" command after 39 days of up time:

Code:
Tasks:  75 total,   1 running,  74 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.8 us,  0.3 sy,  0.0 ni, 96.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:    504480 total,   483184 used,    21296 free,    39516 buffers
KiB Swap:   524284 total,    34844 used,   489440 free.   217072 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  980 root      20   0  367520   6568    988 S   5.0  1.3   3637:47 calibre-se+
 8566 squeeze+  20   0  165916 156400   5120 S   1.7 31.0 136:10.34 squeezebox+
19632 mike      20   0    3384   1392   1012 R   0.7  0.3   0:00.13 top
18777 root      20   0       0      0      0 S   0.3  0.0   0:16.61 kworker/0:2
19609 mike      20   0   11168   1844   1076 S   0.3  0.4   0:00.02 sshd
    1 root      20   0    4328   1328    632 S   0.0  0.3   0:04.36 init
    2 root      20   0       0      0      0 S   0.0  0.0   0:01.36 kthreadd
The pertinent bit here is that the CPU time for calibre is substantially bigger than anything else (> 3600s or 60 hours!). This behaviour is constant over restarts.

For comparison, second place goes to a music server (the "squeezebox" entry) that controls 4 hardware music players and transcodes streamed audio on the fly. It's still racked up only about 4% of the time Calibre has used. In contrast, the Calibre library probably hasn't even been accessed once since the last restart.

Is this expected behaviour? What is likely to be causing it? Currently I'm thinking of uninstalling from the server and going back to using it solely on my desktop.

NB The instructions I followed suggested that xvfb should be installed to "in order to sidestep issues caused by running Calibre in a non-display environment". I didn't do this, reasoning that I could add it later if I saw any "issues" - could I be seeing one now?

Thanks for any help anyone can provide.
TypeC

Last edited by TypeC; 01-05-2016 at 07:39 AM. Reason: Tidy TOP output
TypeC is offline   Reply With Quote
Old 01-05-2016, 07:51 AM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 28,341
Karma: 6652253
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Umm just because you never accessed your calibre server does not mean that no one else accessed it. Look at the server logs. And you dont need xvfb that advice is obsolete.
kovidgoyal is online now   Reply With Quote
Old 01-05-2016, 08:07 AM   #3
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 28,341
Karma: 6652253
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Oh and if you happen to be syncing the calibre library on your server with somewhere else, then everytime the server detects that metadata.db has changed, it has to reload the entire db into memory, which is fairly CPU intensive.
kovidgoyal is online now   Reply With Quote
Old 01-05-2016, 08:54 AM   #4
TypeC
Junior Member
TypeC began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Jan 2016
Location: UK
Device: Nook Glowlight
Thanks for your quick response, Kovid.

Quote:
Originally Posted by kovidgoyal View Post
Umm just because you never accessed your calibre server does not mean that no one else accessed it.
Hopefully, it does - I'm currently the only one in my house who knows how to connect to it (and it's not exposed to the Internet, or shouldn't be).

It's a static library (the timestamp on Metadata.db indicates it was last modified in October).

I've (now) found "calibre-server.log", which appears also not to have been modified since October 27th (I think this may have been when I installed). There's nothing of note in this file, except some errors which I presumably fixed when I was setting it up.

Are there other logs of interest, or a way to get more verbose output in the one I've found?
TypeC is offline   Reply With Quote
Old 01-05-2016, 09:13 AM   #5
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 28,341
Karma: 6652253
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
The logs will be in ~/.config/calibre (assuming whatever you used to setup the server did not change the calibre config directory) named

server_access_log.txt
server_error_log.txt
kovidgoyal is online now   Reply With Quote
Old 01-05-2016, 10:37 AM   #6
eschwartz
Irrational Optimist
eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.
 
eschwartz's Avatar
 
Posts: 16,930
Karma: 66657949
Join Date: Nov 2012
Location: The Beaten Path, USA, Roundworld, This Side of Infinity
Device: Kindle Touch fw5.3.7 (Wifi only)
Quote:
Originally Posted by TypeC View Post
A while ago, I installed Calibre server on a headless 32-bit Atom (x86) system running Ubuntu 14.04.
What version of calibre?
eschwartz is offline   Reply With Quote
Old 01-05-2016, 11:14 AM   #7
TypeC
Junior Member
TypeC began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Jan 2016
Location: UK
Device: Nook Glowlight
Quote:
Originally Posted by eschwartz View Post
What version of calibre?
2.41 (sorry meant to look this up initially and forgot). I originally installed in May 2015, so I've seen the same behaviour between whatever version was current then and this one.

Found the proper logs - thanks, Kovid (they were hidden in /root/.config/calibre - presumably because Ubuntu's Upstart runs as root).

Other than at the point of upgrade (with a couple of other entries relating to /cherrypy/lib/reprconf.pyo having been changed) server_error_log.txt contains only a straightforward (and identical) group of entries for each restart:

Code:
[27/Nov/2015:12:02:04] ENGINE Bus STARTING
[27/Nov/2015:12:02:04] ENGINE Started monitor thread '_TimeoutMonitor'.
[27/Nov/2015:12:02:04] ENGINE Started monitor thread 'Autoreloader'.
[27/Nov/2015:12:02:04] ENGINE Serving on 0.0.0.0:8004
[27/Nov/2015:12:02:05] ENGINE Bus STARTED
and server_access_log.txt has nothing in it between times I know I've accessed the system myself. For example, here are 4 contiguos lines:

Code:
192.168.40.134 - - [27/Oct/2015:17:25:51] "GET /get/thumb_90_120/1377 HTTP/1.1" 200 3259 "http://joggler:8004/browse/category/newest" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:41.0) Gecko/20100101 Firefox/41.0"
192.168.40.134 - - [27/Oct/2015:17:25:51] "GET /get/thumb_90_120/1375 HTTP/1.1" 200 2714 "http://joggler:8004/browse/category/newest" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:41.0) Gecko/20100101 Firefox/41.0"
192.168.40.190 - - [05/Jan/2016:12:30:16] "GET / HTTP/1.1" 200 6109 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:43.0) Gecko/20100101 Firefox/43.0"
192.168.40.190 - - [05/Jan/2016:12:30:16] "GET /static/browse/browse.css HTTP/1.1" 200 7646 "http://joggler:8004/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:43.0) Gecko/20100101 Firefox/43.0"
The whole file looks just like (what I'm sure it is) a log of web requests: loads of "GET" and a few "POST".
TypeC is offline   Reply With Quote
Old 01-05-2016, 11:43 AM   #8
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 28,341
Karma: 6652253
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
First, dont run the server as root. It should be run as the user that has ownership of the library files.

As for your extra CPU usage, normally, I'd be willing to help you track it down, but I am currently in the middle of rewriting the entire server, so the effort would be largely wasted as the new server has no code in common with the old one. So I suggest you just live with the high CPU usage for a few months until calibre 3.0 is released.
kovidgoyal is online now   Reply With Quote
Old 01-05-2016, 12:23 PM   #9
TypeC
Junior Member
TypeC began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Jan 2016
Location: UK
Device: Nook Glowlight
Quote:
Originally Posted by kovidgoyal View Post
First, dont run the server as root. It should be run as the user that has ownership of the library files.
I understand the reasons for running with as few permissions as possible. On Ubuntu though, Upstart is the recommended way of starting tasks with the system and (at least on my system) runs as root. It doesn't appear to have a non-hacky way of changing the user for a specific job, although I think later versions (later than the one on Ubuntu 14.04) have introduced a method to acheive this. Almost all the services on my server are running as root and almost all of the packages were downloaded and initialised using scripts from the Ubuntu official repo. It is a custom distro though (for my device) and may not have been built to normal standards.

Other than the fact it is a server (which can potentially interact with the outside world), do you have a specific reason for giving this warning?

Quote:
Originally Posted by kovidgoyal View Post
As for your extra CPU usage, normally, I'd be willing to help you track it down, but I am currently in the middle of rewriting the entire server, so the effort would be largely wasted as the new server has no code in common with the old one.
Under the circumstances, that's quite understandable and perfectly fine, of course. I'm very grateful to you for providing this wonderful software at all.
TypeC is offline   Reply With Quote
Old 01-05-2016, 12:52 PM   #10
TypeC
Junior Member
TypeC began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Jan 2016
Location: UK
Device: Nook Glowlight
I think I may have fixed this. The blog post I linked to earlier which I followed to get things up and running included "--auto-reload" on the command-line arguments for calibre-server.

If I understand the Calibre docs, there's not a lot of point in that. I removed it and now Calibre is sitting at about 0.3% CPU instead of ~5.5%.

I'll look out in a couple of days, but hopefully that will be it!

Thanks again for the help.

Last edited by TypeC; 01-05-2016 at 12:53 PM. Reason: typo
TypeC is offline   Reply With Quote
Old 01-05-2016, 01:01 PM   #11
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 28,341
Karma: 6652253
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Yeah auto reload is a definite no-no in production. It is meant for development.

Running internet facing software as root is pretty dangerous, but other than that, calibre does not care.
kovidgoyal is online now   Reply With Quote
Old 01-05-2016, 01:57 PM   #12
TypeC
Junior Member
TypeC began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Jan 2016
Location: UK
Device: Nook Glowlight
Sorry to continue this in a slightly off-topic manner, but it turns out it is relatively straightforward to run Upstart jobs as non-privileged users (even though it appears to be uncommon), even if you are using an older version lacking the setuid / setgid settings.

The script has an "exec" statement for the daemon executable: I just put "sudo -u <user>" between "exec" and the name of the executable and the process owner is now showing as "<user>". I don't know whether this has any downsides, but everything I can think of trying appears to be working correctly.

Hopefully this will help anyone else finding this thread in a similar position to me.

(Aside - I've tried three times to correct the original tutorial post - so fewer people *will* be in the same position, but my comments keep being rejected as spam.)
TypeC is offline   Reply With Quote
Old 01-05-2016, 02:46 PM   #13
eschwartz
Irrational Optimist
eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.
 
eschwartz's Avatar
 
Posts: 16,930
Karma: 66657949
Join Date: Nov 2012
Location: The Beaten Path, USA, Roundworld, This Side of Infinity
Device: Kindle Touch fw5.3.7 (Wifi only)
Wow...

I think I prefer Wikis. Too many tutorials are barely good enough to give you inspiration for formulating your own solution.
eschwartz is offline   Reply With Quote
Reply

Tags
linux ubuntu, server

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Calibre not using all CPU cores. mufler Calibre 3 10-07-2013 09:59 AM
Calibre and CPU nlundberg Calibre 7 02-15-2013 07:45 AM
is it possible to make calibre just use one CPU? booksonthemove Calibre 1 02-20-2010 05:55 PM
iLiad ipdf with CPU Throttling yokos iRex Developer's Corner 61 08-26-2008 08:10 PM
iLiad overclock or change cpu? joblack iRex Developer's Corner 6 11-11-2007 01:27 AM


All times are GMT -4. The time now is 12:58 PM.


MobileRead.com is a privately owned, operated and funded community.