CalibreDB add and Temporary file descriptors filling up disk

[seru1us]

Editing to add images as links for everyone's reference.

Hey, everyone-

First post here so forgive me if this is the wrong place to post, tried searching for something similar.

Essentially, I'm using "calibredb add" to add quite a few ebooks. Everything seems to work fine, except for the fact that calibredb seems to keep open file descriptors to the temp folder, which fills it up quickly. Has anyone seen something similar, or have a workaround besides restarting the application and clearing out the descriptors? Here is some information of my system:

calibredb version: (calibre 3.22.1)
uname -a: Linux drax.packet.farm 4.13.16-2-pve #1 SMP PVE 4.13.16-47 (Mon, 9 Apr 2018 09:58:12 +0200) x86_64 x86_64 x86_64 GNU/Linux

Here is a screenshot of the folder size (/mnt/ramdisk is used for the temp folder, note it's current size)

as well as the du -sh inside the /mnt/ramdisk

Now, per this link, I tracked down the disk usage to open file descriptors. Which, to be quite frank, there seem to be a ton of from calibredb. See this for the results of lsof | grep '(deleted)' to see this behavior.


As well as a count for how many open file descriptors there actually are


Based on this info, for some reason the file descriptors are staying open for long enough to fill up the disk and prevent the calibredb command from finishing.

This behavior continues until the disk is full, and then the calibredb breaks.

Anyone have an idea of how to proceed with this? Thanks!

[kovidgoyal]

Use CALIBRE_TEMP_DIR to tell calibre to use a directory on a disk that has enough space for copies of all the files you are adding. Or add in batches.

And note that file descriptors do not fill up a disk, files do. A file descriptor is simply a small structure, typically a few bytes in size stored inthe kernel.

[seru1us]

Originally Posted by kovidgoyal

Use CALIBRE_TEMP_DIR to tell calibre to use a directory on a disk that has enough space for copies of all the files you are adding. Or add in batches.

And note that file descriptors do not fill up a disk, files do. A file descriptor is simply a small structure, typically a few bytes in size stored inthe kernel.

Thanks for the reply!

So for a very large folder structure, I'm guessing I should write a script to run through each folder and run calibredb add on each one, as opposed to running it on the entire parent directory.

I think the main issue here is that the file descriptor stays open for much longer after the file is deleted, so the space in the folder is still allocated. Regardless, I'll write up a script to go with this workaround. Thanks again!

[seru1us]

For anyone wondering, this is how I ended up adding an enormous directory using calibredb, so far its running well without filling up the disk.

find YOUR_HUGE_FOLDER -maxdepth 1 -mindepth 1 -type d -exec bash -c 'echo "$0" && xvfb-run calibredb add -r "$0"/* --library-path YOUR_LIBRARY_PATH' {} \;

just change YOUR_HUGE_FOLDER and YOUR_LIBRARY_PATH to the target path and the library folder.

Feel free to change the "maxdepth" to how far into the folder you want to dive.

edit: made a few changes.