Why calibre insists on using its own directory structure with a copy of your books
(Personally, I feel calibre does a much better job of keeping track of my books, and leave the
only copies in calibre.)
Note that you can store the library anywhere you want, move it, or create a second library, see the Switch/Create Library dialog in the Library dropdown:
...
As stated above, the memory error is because calibre copies the ebook into $TMP while adding it, before finally adding the copy into the Calibre Library. Additionally, it needs to keep track of the book with its metadata and a few other things... all in RAM.
Adding many books at a time means all those books get copied into $TMP together, if you add small chunks at a time then it is much more manageable.
Some systems use a ramdisk for $TMP, which means that the ebook copies also take up RAM.
Regardless, calibre does a lot of work while adding, and eats up more RAM the more books you add at a time. You are advised to not try adding several thousand books at a time...