Help! Calibre is Eating All My Memory!

[eschwartz]

Why calibre insists on using its own directory structure with a copy of your books
(Personally, I feel calibre does a much better job of keeping track of my books, and leave the only copies in calibre.)

Note that you can store the library anywhere you want, move it, or create a second library, see the Switch/Create Library dialog in the Library dropdown:


...

As stated above, the memory error is because calibre copies the ebook into $TMP while adding it, before finally adding the copy into the Calibre Library. Additionally, it needs to keep track of the book with its metadata and a few other things... all in RAM.
Adding many books at a time means all those books get copied into $TMP together, if you add small chunks at a time then it is much more manageable.

Some systems use a ramdisk for $TMP, which means that the ebook copies also take up RAM.
Regardless, calibre does a lot of work while adding, and eats up more RAM the more books you add at a time. You are advised to not try adding several thousand books at a time...

[Joanna]

Write down all the collections that you've got, including their sizes and on which hard drive they're located.

Think of which of them make sense, and which could be better divided/combined (if you have enough space on one of the drives).

Choose one of those collections for the start, and create a Calibre library on the same drive where the collection is (this way you can be pretty sure you won't run out of space). If you want to create the library on another drive that the one the files are currently stored on, make space first and copy the folder with collection there (because this way you won't end up, missing space for the books).

Add files in small batches (10-100 files, depending on the size of the file and the amount of free space on the drive), and delete them after adding (end empty the recycle bin).

When you get a bigger drive, you'll be able to combine some or all libraries into one (which is pretty straightforward). That's what I would recommend: I have one big library with virtual libraries in it, and for me, it seems like a much better solution than a couple of separate libraries (at least for now... I like playing with Calibre a lot and keep discovering new solutions and possibilities ).

Do you have back-up of all your files? If not, I would generally recommend buying a new big hard drive / a couple of smaller ones, to have everything in two copies (not because of Calibre, just for safety reasons).

[glennWilliams]

Originally Posted by kovidgoyal

Adding books requires approximately three times the disk space used by the actual book files during the adding process. SO either add in small batches or use an environment variable to move the calibre temp dir to a disk with more space.

hi

can i just get clarification here?

I see calibre adding to the TMP folder during the add process. are you saying that it only adds to the actual library at the end of the add process, so before the end the books are in two places; original and tmp. OR is it adding to TMP, then a process adding from TMP to library an only deleting TMP at end?

im asking if I can start deleting TMP files during the add process to free up some space.

if the first way, does this also mean that if I quite an add part way through that files will not be in the library, still only in temp?

if the second way, are files left in TMP folder (even though they are already in library) because of file locks or something?

is it not possible to get from original location, move to temp, then add to library and delete temp once added. rather than waiting until end of add.

I just want to know what the process is at it will change how I add books in future. (ive only just started using app this last month)

Im interesting in getting involved in coding too. have just grabbed code to have a look through.

I know u also have a 'auto add from folder' feature but this also seems to use a lot of disk space if the auto add folder has a large number of books added to it.

thanks

glenn

[kovidgoyal]

https://github.com/kovidgoyal/calibr...re/gui2/add.py

[glennWilliams]

hi ya.

in addition to my post.

looking at the add I have in progress. it looks like in my tmp folder I now have a copy of ALL the books from the folder I am adding from. is this whats happening: workers building a tmp folder from original, far ahead of where the add to library is, then a process working through the tmp folder adding from there to library? once the tmp folder has copy of all books from original is the original folder no longer accessed? ie I could move/delete the original folder part way through the add process as calibre is now working only from the TMP folder to the library?

thx