For anyone wondering, this is how I ended up adding an enormous directory using calibredb, so far its running well without filling up the disk.
Code:
find YOUR_HUGE_FOLDER -maxdepth 1 -mindepth 1 -type d -exec bash -c 'echo "$0" && xvfb-run calibredb add -r "$0"/* --library-path YOUR_LIBRARY_PATH' {} \;
just change YOUR_HUGE_FOLDER and YOUR_LIBRARY_PATH to the target path and the library folder.
Feel free to change the "maxdepth" to how far into the folder you want to dive.
edit: made a few changes.