Memory leak in calibre-server?

[mondai]

Possibly there is a memory leak in calibre-server.

calibre-server runs on an unused netbook with 64-bit Linux, dual core, 2GB RAM and 1GB swap space. X is not running. Server load is very low (a few ebook readers).

After some days calibre-server consumes all available memory and Linux begins to trash between RAM and swap. The machine is unusable. Restarting calibre-server solves this problem immediately.

[PeterT]

What version of calibre and is it installed from the official calibre site or from your distribution's repository?

[mondai]

Originally Posted by PeterT

What version of calibre and is it installed from the official calibre site or from your distribution's repository?

It's the latest version 1.41.0 "Linux 64-bit binary" directly from http://download.calibre-ebook.com.

OS is latest Lubuntu 14.04 64-bit.

[kovidgoyal]

I cannot reproduce this. You will need to figure out what is leaking exactly. Monitor the memory usage of the process and make some requests from the server, see which requests are causing the increase in memory consumption, and note that the server is written in python which automatically manages memory so monitoring memory is not completely straightforward as python will not always relinquish memory to the OS completely, so to detect a leak you need to make the same request repeatedly and check that the memory usage climbs by approximately the same amount each time.

[mondai]

The virtual memory size of calibre-server is very high: 2 GB. That is equal to the complete physical memory. Take a look at the output of memstat below.

Could it be that there is a problem with the CPU and the OS? The CPU is an Intel N570 Atom Processor. "Max. Memory Size" is 2 GB. Therefore it makes no sense to use additional swap space (except for hibernation).
Intel Atom Processor N570


Another strange behavior is loading a lot of X-related libraries, but X is not running.

2394708k: PID  2047 (/opt/calibre-1.41/bin/calibre-server)
  39168k(      0k): /opt/calibre-1.41/lib/libicudata.so.49 2047 2053 2047 2053 2047 2053
  29324k(  21988k): /opt/calibre-1.41/lib/libQtWebKit.so.4 2047 2053 2047 2053 2047 2053
  15616k(  10944k): /opt/calibre-1.41/lib/libQtGui.so.4 2047 2053 2047 2053 2047 2053 2047 2053
  12996k(   6620k): /opt/calibre-1.41/lib/python2.7/site-packages/PyQt4/QtGui.so 2047 2053 2047 2053 2047 2053
   8940k(   4348k): /opt/calibre-1.41/lib/libQtXmlPatterns.so.4 2047 2053 2047 2053 2047 2053 2047 2053
   7140k(   2948k): /opt/calibre-1.41/lib/libQtCore.so.4 2047 2053 2047 2053 2047 2053
   7016k(   2136k): /opt/calibre-1.41/lib/python2.7/site-packages/PyQt4/QtCore.so 2047 2053 2047 2053 2047 2053 2047 2053
   6972k(   2036k): /opt/calibre-1.41/lib/libMagickCore.so.5 2047 2053 2047 2053 2047 2053
   6320k(   2120k): /opt/calibre-1.41/lib/libicui18n.so.49 2047 2053 2047 2053 2047 2053 2047 2053
   6080k(   1688k): /opt/calibre-1.41/lib/libcrypto.so.1.0.0 2047 2053 2047 2053 2047 2053 2047 2053
   6080k(   1480k): /opt/calibre-1.41/lib/libpython2.7.so.1.0 2047 2053 2047 2053 2047 2053
   5740k(   1220k): /opt/calibre-1.41/lib/python2.7/site-packages/lxml/etree.so 2047 2053 2047 2053 2047 2053
   5700k(   1460k): /opt/calibre-1.41/lib/libicuuc.so.49 2047 2053 2047 2053 2047 2053
   5476k(   1316k): /opt/calibre-1.41/lib/libQtNetwork.so.4 2047 2053 2047 2053 2047 2053
   5472k(   1304k): /opt/calibre-1.41/lib/libxml2.so.2 2047 2053 2047 2053 2047 2053 2047 2053
   5248k(   1120k): /opt/calibre-1.41/lib/libMagickWand.so.5 2047 2053 2047 2053 2047 2053
   5240k(   1128k): /opt/calibre-1.41/lib/libglib-2.0.so.0 2047 2053 2047 2053 2047 2053
   5136k(    968k): /opt/calibre-1.41/lib/libstdc++.so.6 2047 2053 2047 2053 2047 2053
   4940k(    612k): /opt/calibre-1.41/lib/python2.7/site-packages/PyQt4/QtNetwork.so 2047 2053 2047 2053 2047 2053
   4848k(    600k): /opt/calibre-1.41/lib/python2.7/lib-dynload/unicodedata.so 2047 2053 2047 2053 2047 2053
   4824k(    696k): /opt/calibre-1.41/lib/libsqlite3.so.0 2047 2053 2047 2053 2047 2053 2047 2053
   4616k(    504k): /opt/calibre-1.41/lib/libQtDBus.so.4 2047 2053 2047 2053 2047 2053 2047 2053
   4612k(    380k): /opt/calibre-1.41/lib/python2.7/site-packages/PyQt4/QtWebKit.so 2047 2053 2047 2053 2047 2053 2047 2053
   4588k(    460k): /opt/calibre-1.41/lib/libgcrypt.so.11 2047 2053 2047 2053 2047 2053
   4560k(    432k): /opt/calibre-1.41/lib/libtiff.so.5 2047 2053 2047 2053 2047 2053
   4552k(    384k): /opt/calibre-1.41/lib/libssl.so.1.0.0 2047 2053 2047 2053 2047 2053 2047 2053
   4480k(    360k): /opt/calibre-1.41/lib/libQtSvg.so.4 2047 2053 2047 2053 2047 2053
   4480k(    272k): /opt/calibre-1.41/lib/python2.7/site-packages/PyQt4/QtXml.so 2047 2053 2047 2053 2047 2053
   4476k(    276k): /opt/calibre-1.41/lib/python2.7/site-packages/PyQt4/QtDesigner.so 2047 2053 2047 2053 2047 2053
   4428k(    244k): /opt/calibre-1.41/lib/python2.7/site-packages/PyQt4/QtOpenGL.so 2047 2053 2047 2053 2047 2053
   4404k(    276k): /opt/calibre-1.41/lib/liblcms2.so.2 2047 2053 2047 2053 2047 2053
   4404k(    268k): /opt/calibre-1.41/lib/python2.7/site-packages/apsw.so 2047 2053 2047 2053 2047 2053
   4396k(    276k): /opt/calibre-1.41/lib/libQtXml.so.4 2047 2053 2047 2053 2047 2053
   4364k(    188k): /opt/calibre-1.41/lib/python2.7/site-packages/PyQt4/QtDBus.so 2047 2053 2047 2053 2047 2053
   4360k(    248k): /opt/calibre-1.41/lib/libjpeg.so.8 2047 2053 2047 2053 2047 2053
   4348k(    236k): /opt/calibre-1.41/lib/libdbus-1.so.3 2047 2053 2047 2053 2047 2053
   4328k(    224k): /opt/calibre-1.41/lib/libxslt.so.1 2047 2053 2047 2053 2047 2053
   4296k(    144k): /opt/calibre-1.41/lib/python2.7/site-packages/PyQt4/QtXmlPatterns.so 2047 2053 2047 2053 2047 2053
   4296k(    112k): /opt/calibre-1.41/lib/python2.7/lib-dynload/_io.so 2047 2053 2047 2053 2047 2053
   4272k(    152k): /opt/calibre-1.41/lib/libexpat.so.1 2047 2053 2047 2053 2047 2053
   4256k(    152k): /opt/calibre-1.41/lib/libpng14.so.14 2047 2053 2047 2053 2047 2053
   4240k(    104k): /opt/calibre-1.41/lib/python2.7/lib-dynload/_ctypes.so 2047 2053 2047 2053 2047 2053
   4224k(    112k): /opt/calibre-1.41/lib/python2.7/site-packages/sip.so 2047 2053 2047 2053 2047 2053
   4200k(     80k): /opt/calibre-1.41/lib/python2.7/site-packages/PyQt4/QtSvg.so 2047 2053 2047 2053 2047 2053
   4196k(     68k): /opt/calibre-1.41/lib/python2.7/lib-dynload/datetime.so 2047 2053 2047 2053 2047 2053
   4192k(     80k): /opt/calibre-1.41/lib/libz.so.1 2047 2053 2047 2053 2047 2053
   4192k(     56k): /opt/calibre-1.41/lib/python2.7/lib-dynload/_socket.so 2047 2053 2047 2053 2047 2053
   4188k(     76k): /opt/calibre-1.41/lib/libexslt.so.0 2047 2053 2047 2053 2047 2053
   4176k(     72k): /opt/calibre-1.41/lib/python2.7/lib-dynload/cPickle.so 2047 2053 2047 2053 2047 2053
   4176k(     40k): /opt/calibre-1.41/lib/python2.7/lib-dynload/itertools.so 2047 2053 2047 2053 2047 2053
   4164k(     60k): /opt/calibre-1.41/lib/libbz2.so.1 2047 2053 2047 2053 2047 2053
   4160k(     48k): /opt/calibre-1.41/lib/libicuio.so.49 2047 2053 2047 2053 2047 2053 2047 2053
   4160k(     40k): /opt/calibre-1.41/lib/python2.7/lib-dynload/pyexpat.so 2047 2053 2047 2053 2047 2053
   4156k(     52k): /opt/calibre-1.41/lib/python2.7/site-packages/calibre/plugins/magick.so 2047 2053 2047 2053 2047 2053
   4144k(     40k): /opt/calibre-1.41/lib/python2.7/lib-dynload/_elementtree.so 2047 2053 2047 2053 2047 2053
   4144k(     32k): /opt/calibre-1.41/lib/python2.7/lib-dynload/operator.so 2047 2053 2047 2053 2047 2053
   4144k(     32k): /opt/calibre-1.41/lib/python2.7/lib-dynload/math.so 2047 2053 2047 2053 2047 2053
   4144k(     32k): /opt/calibre-1.41/lib/python2.7/lib-dynload/array.so 2047 2053 2047 2053 2047 2053
   4140k(     28k): /opt/calibre-1.41/lib/python2.7/site-packages/PyQt4/QtTest.so 2047 2053 2047 2053 2047 2053
   4140k(     28k): /opt/calibre-1.41/lib/python2.7/site-packages/calibre/plugins/icu.so 2047 2053 2047 2053 2047 2053
   4140k(     28k): /opt/calibre-1.41/lib/python2.7/lib-dynload/_struct.so 2047 2053 2047 2053 2047 2053
   4140k(     20k): /opt/calibre-1.41/lib/python2.7/lib-dynload/_csv.so 2047 2053 2047 2053 2047 2053
   4136k(     32k): /opt/calibre-1.41/lib/python2.7/lib-dynload/_ssl.so 2047 2053 2047 2053 2047 2053
   4136k(     32k): /opt/calibre-1.41/lib/python2.7/lib-dynload/_json.so 2047 2053 2047 2053 2047 2053
   4136k(     24k): /opt/calibre-1.41/lib/python2.7/lib-dynload/_collections.so 2047 2053 2047 2053 2047 2053
   4132k(     28k): /opt/calibre-1.41/lib/libffi.so.4 2047 2053 2047 2053 2047 2053
   4132k(     20k): /opt/calibre-1.41/lib/python2.7/lib-dynload/zlib.so 2047 2053 2047 2053 2047 2053
   4132k(     20k): /opt/calibre-1.41/lib/python2.7/lib-dynload/strop.so 2047 2053 2047 2053 2047 2053
   4132k(     20k): /opt/calibre-1.41/lib/python2.7/lib-dynload/select.so 2047 2053 2047 2053 2047 2053
   4128k(     16k): /opt/calibre-1.41/lib/python2.7/lib-dynload/time.so 2047 2053 2047 2053 2047 2053
   4128k(     16k): /opt/calibre-1.41/lib/python2.7/lib-dynload/cStringIO.so 2047 2053 2047 2053 2047 2053
   4124k(     12k): /opt/calibre-1.41/lib/python2.7/lib-dynload/_heapq.so 2047 2053 2047 2053 2047 2053
   4120k(     16k): /opt/calibre-1.41/lib/python2.7/lib-dynload/_locale.so 2047 2053 2047 2053 2047 2053
   4120k(     16k): /opt/calibre-1.41/lib/python2.7/lib-dynload/binascii.so 2047 2053 2047 2053 2047 2053
   4116k(     12k): /opt/calibre-1.41/lib/python2.7/lib-dynload/_random.so 2047 2053 2047 2053 2047 2053
   4116k(     12k): /opt/calibre-1.41/lib/python2.7/lib-dynload/_hashlib.so 2047 2053 2047 2053 2047 2053
   4116k(     12k): /opt/calibre-1.41/lib/python2.7/lib-dynload/_functools.so 2047 2053 2047 2053 2047 2053
   4116k(     12k): /opt/calibre-1.41/lib/python2.7/lib-dynload/fcntl.so 2047 2053 2047 2053 2047 2053
   4116k(     12k): /opt/calibre-1.41/lib/libgpg-error.so.0 2047 2053 2047 2053 2047 2053
   4116k(     12k): /opt/calibre-1.41/lib/libcalibre-launcher.so 2047 2053 2047 2053 2047 2053
   4112k(      8k): /opt/calibre-1.41/lib/python2.7/site-packages/calibre/plugins/speedup.so 2047 2053 2047 2053 2047 2053
   4112k(      8k): /opt/calibre-1.41/lib/python2.7/lib-dynload/grp.so 2047 2053 2047 2053 2047 2053
   4112k(      8k): /opt/calibre-1.41/lib/python2.7/lib-dynload/_bisect.so 2047 2053 2047 2053 2047 2053
   4108k(      4k): /opt/calibre-1.41/lib/python2.7/site-packages/PyQt4/Qt.so 2047 2053 2047 2053 2047 2053
   4108k(      4k): /opt/calibre-1.41/lib/python2.7/lib-dynload/future_builtins.so 2047 2053 2047 2053 2047 2053
   2096k(     40k): /opt/calibre-1.41/lib/ImageMagick-6.7.6/modules-Q16/coders/jpeg.so 2047
     20k(      4k): /opt/calibre-1.41/bin/calibre-server 2047 2053 2047 2053 2047 2053

[kovidgoyal]

memory usage and memory leaks are very different things. Which one are you asking about?

[mondai]

Hmm. I'm not a developer. I only noticed that my machine runs out of memory caused by a simple webserver. "Memory leak" or "extremely memory consumption"? Both is not OK.

My workaround is restarting calibre-server each morning by a cron job. That's crazy but it works...