"Too many open files" errors

[_jal]

Hi all,

I'm using Calibre 1.9 (now, although this issue has been present in past versions) on Mac OS X 10.0 (although again, this problem existed on 10.8.3).

Nearly immediately upon starting Calibre, I get this error

calibre, version 1.9.0
ERROR: Unhandled exception: <b>IOError</b>:[Errno 24] Too many open files: '/Users/jal/Library/Preferences/calibre/gui.json'

Traceback (most recent call last):
  File "site-packages/calibre/gui2/metadata/single_download.py", line 1067, in accept
  File "site-packages/calibre/utils/config.py", line 397, in __setitem__
  File "site-packages/calibre/utils/config.py", line 341, in commit
  File "site-packages/calibre/utils/lock.py", line 113, in __enter__
IOError: [Errno 24] Too many open files: '/Users/jal/Library/Preferences/calibre/gui.json'

This actually started some time back after substantial use; now it will happen after one or two uses of the "Edit Metadata" function (for instance).

Techie notes: I increased the maxfiles limit (via launchctl, the OSX equivalent of other unixen's ulimit), which gave a little headroom and then choked again; I poked around a bit in the code, but I didn't see any obvious file handle leaks, but then again, it is a completely new codebase to me.


My library currently has 3900-odd books; is this a case of the wrong hammer for the job - is Calibre not expected to handle libraries of this size??

[DoctorOhh]

Originally Posted by _jal

My library currently has 3900-odd books; is this a case of the wrong hammer for the job - is Calibre not expected to handle libraries of this size??

Calibre is designed to scale to accommodate quite large libraries. My library is around 13,000 books and there are no issues. Other have reported calibre handling their massive libraries of 50 - 100 thousand books.

Good Luck.

[BetterRed]

There's a program called Tasks Explorer for OS/X that will tell you what files are opened by what processes

BR

[kovidgoyal]

You've probably enabled the google images download plugin, which uses a full webkit instance and requires lots of file handles. However, I dont see how it's possible for it to leak file handles, since it runs in a separate process which is shutdown after the download completes.

[stackm]

FYI, just wanted to add I have been seeing this error for a few weeks on several different calibre versions. Currently have 1.24 and Mac OS Mavericks. Didn't see it before upgrading OS to Mavericks in October. 5000 ebooks.

calibre, version 1.24.0
ERROR: Unhandled exception: <b>Exception</b>:Failed to open u'/Users/stackm/Calibre Library/x (7512)/cover.jpg' with error: [Errno 24] Too many open files: u'/Users/stackm/Calibre Library/x (7512)/cover.jpg'

Traceback (most recent call last):
File "site-packages/calibre/gui2/cover_flow.py", line 109, in image
File "site-packages/calibre/gui2/library/models.py", line 668, in cover
File "site-packages/calibre/db/legacy.py", line 332, in cover
File "site-packages/calibre/db/cache.py", line 607, in cover
File "site-packages/calibre/db/cache.py", line 57, in call_func_with_lock
File "site-packages/calibre/db/cache.py", line 648, in copy_cover_to
File "site-packages/calibre/db/backend.py", line 1242, in copy_cover_to
Exception: Failed to open u'/Users/stackm/Calibre Library/x(7512)/cover.jpg' with error: [Errno 24] Too many open files: u'/Users/stackm/Calibre Library/x (7512)/cover.jpg'


It doesn't happen immediately, but I will see it after leaving it open for a few hours or overnight.

I did increase file descriptors on the OS. Do not have Google images plugin.

Calibre will refuse to exit, and keep popping up a box with the error. When I click OK it gives the error for a different item. So there is no way to restart it without killing the Calibre process using Activity Monitor.

Will experiment with Tasks Explorer.

[kovidgoyal]

You can always track open files with the lsof command line tool to see what files are being left open, like this

lsof | grep calibre

Run it repeatedly and see what files are being leaked.

A bit of googling will probably find some GUI versions you can use as well.