Help! Calibre is Eating All My Memory!

kingseesar · 06-18-2016, 01:05 PM

Hi everyone!

First off, apologies if this has been addressed before. I did several searches in the forum and haven't found my specific situation and desperately need some help:

I just got Calibre two days ago after adding my epubs from old hard drives off another three computers. There's a TON of books, and I've been trying to add them to my library chunk or two at a time. I don't have much memory with which to start, to be fair, but I had about 19GB or so. Should be plenty for starters. So I started using the "Add Books" feature to organize my files. I hit the button and Calibre tells me it's going to take a while (read as the progress bar was moving very slowly and took almost 30 mins to get to 3%), so I went to bed, leaving my computer to finish importing books.

When I woke up, I got a memory error message and found that my hard drive has no memory left at all! Calibre has used up 19GB of memory importing books! Is it duplicating them somewhere? Is it expanding metadata somewhere that's doubling or tripling the size of my files? What can I do to revert this nightmare? I thought Calibre was just going to make note of where my books were located on my computer and wouldn't be using any memory to add them to my library. I have about 2.5TB of books that I wanted to add, but if I can't do it in Calibre without doubling my memory use, there's no point in using it...

Any help or suggestions will be greatly appreciated! Thanks in advance! Cheers!
Will

kingseesar · 06-18-2016, 01:10 PM

Slight update - When I clicked "OK" to exit the error message, 3.03GB of memory mysteriously returned to my hard drive. I still have no idea what's going on, either...

Thanks again!

kovidgoyal · 06-18-2016, 01:29 PM

Adding books requires approximately three times the disk space used by the actual book files during the adding process. SO either add in small batches or use an environment variable to move the calibre temp dir to a disk with more space.

jhowell · 06-18-2016, 01:43 PM

Quote:

Originally Posted by kingseesar

I thought Calibre was just going to make note of where my books were located on my computer and wouldn't be using any memory to add them to my library.

Calibre copies the actual book files into its own library directory structure. The originals will be left in place.

kingseesar · 06-18-2016, 02:16 PM

Thanks Kovidgoyal and Jhowell! Cheer for getting back to me so fast!

I'm rather surprised that Calibre doesn't just make note of where the books are and then gather metadata and open them from their current location. Sounds like I can't use Calibre, then, which is too bad, as I quite liked the interface and features, but I have so many books and comics on my hard drives and I certainly don't have the thousands of free gig space necessary to add the books to my Calibre library, which was the point of using it. And that's not an exaggeration. I have almost 2TB of comics alone, then there's the epubs I've been collecting for years, which is almost another 1TB, so Calibre just isn't feasible, sadly.

Oh well... Thank you so much for your responses! I'll figure something out. Cheers!
Will

DaltonST · 06-18-2016, 03:03 PM

@kingseesar:

Calibre > Preferences > Add Books > Automatic Adding has the ability to automatically delete the source books being added after they have been added.

DaltonST

theducks · 06-18-2016, 03:07 PM

Quote:

Originally Posted by kingseesar

Thanks Kovidgoyal and Jhowell! Cheer for getting back to me so fast!

I'm rather surprised that Calibre doesn't just make note of where the books are and then gather metadata and open them from their current location. Sounds like I can't use Calibre, then, which is too bad, as I quite liked the interface and features, but I have so many books and comics on my hard drives and I certainly don't have the thousands of free gig space necessary to add the books to my Calibre library, which was the point of using it. And that's not an exaggeration. I have almost 2TB of comics alone, then there's the epubs I've been collecting for years, which is almost another 1TB, so Calibre just isn't feasible, sadly.

Oh well... Thank you so much for your responses! I'll figure something out. Cheers!
Will

Do a small group. Then offload those.
Repeat

(better yet: Get a bigger drive. Most sell with a clone the drive softwar package available. CloneZilla will do dual boot drives, where Acronis will not)

kingseesar · 06-18-2016, 03:35 PM

@DaltonST : So... I have the option of deleting the original copies of the books once they have been added to Calibre, then? I'm not sure that works, either, only because that means I have to consolidate all my books onto one drive and, as before, I don't have that kind of space. I'd need a 3TB drive or bigger to do that, and I don't have that kind of money at the moment. Still, it's good to know that's a possibility. I've already added over 3000 books to Calibre's library, but I think they are all from the same drive where the library is stored. I assume there's a way to move the library to another drive, somehow? Maybe if I can get a new bigger drive, I can put them all there? I have four external drives and one internal drive, all with books/comics on them at the moment, thus I was hoping for something that would just make a note of where they are located and grab them as needed, but I'll keep this in mind for sure. Thank you!

@theducks : and you are saying that the memory usage is temporary when adding books to the library. @jhowell above suggests that it duplicates the book in its entirety and even if Calibre does only use triple the memory temporarily, that only drops to double the memory used when adding a book. Is that correct? I'm happy to do small groups, if that helps, but it sounds like it won't in the long run.

And again, I'd love to get a bigger drive, but that's out of my spending range at the moment. I've got enough space on my drives for now for daily use (around 400GB total between them all) and I do a lot of audio and video editing, so I need to keep that semi free, but once I get the funds, absolutely, I'll upgrade like a fiend!

Thanks again to everyone for your responses! Cheers! This is all helpful. I certainly have a better understanding of how the program works and generates the library information; sadly, it's still not sounding like there's a feasible solution, yet, though, without a much, much, much bigger hard drive. I'll keep thinking on it and see what I can sort out. Thanks again! You all rock! \m/

BetterRed · 06-18-2016, 04:38 PM

@kingseesar - some people use calibre as a catalog only, and leave the book files where they are. Hopefully one of them will drop by and give some guidance. But here are some thoughts

Firstly have a browse through this ==>> Calibre command line interface

If you want calibre to extract metadata from the format files (EPUB, PDF etc) as it adds (i.e. you've checked Preferences->Add books->Read metadata from file contents...) then you will have to add the format files - but you do not have to keep them in the library. Once the book is added you can remove the format files - you should use calibre's Remove Formats feature to do this rather than your file manager. Using the file manager will result in the calibre library's database and its folder/file contents being inconsistent which is best avoided,

Or, you could use the ebook-meta command to extract the metadata from the existing files into OPF files and add those to the calibre GUI. Calibre's going to retain an OPF anyway, they are used by its database restore feature.

On Windows you could add a shortcut file to a book that references the current location. Alternatively you could put a hyperlink to the existing format file (folder) into the Comments column (or a similar custom column) or into an Arbitrary Identifier. The advantage of using shortcuts is that if you rename or move a shortcut target, Windows will try to repair any LNK files that refer to it - providing you haven't disabled the DLT Service. AFAIK there is no similar feature to repair broken file system URIs - on Windows, OSX or Linux.

Using shortcut files or URLs (file:\\\blah de blah) precludes the use of some of the GUI features - particularly those relating to devices. However many features are also available from the command line - eg conversion and the aforementioned ebook-meta command. The calibredb command also has many useful features.

One of the main attributes of calibre is its flexibility - i.e. there are many ways to skin a cat, other than dragging it backwards through a keyhole

Added - have you taken the time to watch the Video Tutorials. I found them very useful when I started to use calibre - I'm someone who despises video 'instructions'. Watch a segment, then practise it on a test library, if it doesn't work as expected - re-wind/re-play/re-practise.

BR

DaltonST · 06-18-2016, 04:54 PM

Quote:

@DaltonST : So... I have the option of deleting the original copies of the books once they have been added to Calibre, then? I'm not sure that works, either, only because that means I have to consolidate all my books onto one drive and, as before, I don't have that kind of space.

Who said that you can have only a single Calibre library?

What makes you think that all Calibre libraries have to be on the same drive?

A different physical Calibre library based on broad differences might make sense. Comics do not need to be in the same physical library as Non-Fiction or as Fiction.

DaltonST

kingseesar · 06-18-2016, 05:15 PM

@BetterRed : Okay. I think that makes sense. If I can get my head around that, then making a catalogue only would suffice, except I do need to be able to read the epubs using Calibre (or some reader). The other formats are not a problem. Cool! I'll look more into this. Cheer!

@DaltonST : I thought that Calibre was duplicating my files somewhere of it's choice and I wasn't able to find in the preferences how to reassign the drive or directory into which Calibre puts the copied files, so I thought it was set, and thus the library had to be on the same drive as Calibre. obviously, I'm wrong, so I'll try and figure that out, too. As for separate libraries, potentially, as there are a few things I collect/read/research, but they will still be huge libraries. My Lovecraft-related collection is well over 5000 entires, and my comic collection is well over 25,000. The Smaller stuff would definitely work though (the Ethnomusicology (2000) and the Gothic Studies (900) and then the stragglers...) Nice! I'll see about splitting things up, then, if that works. Perhaps the problem is that I was just hoping to put everything in one place and not have to relocate anything, so I apologise if I'm just being lazy.

Excellent! Thanks again everyone! Very helpful responses! Cheers! I really appreciate the assistance!
Will

BetterRed · 06-18-2016, 05:25 PM

@kingseesar - The calibre viewer is a discrete program - ebook-viewer - you don't need the library manager to use it, same deal with the calibre editor, it's ebook-edit. I have them both in my OS file associations for EPUB - the viewer as the default.

I keep separate libraries for books, comics, magazines/journals, picture albums and media (video and audio clips) - mainly because they have quite different metadata data elements, or I use the same metadata items differently - eg in one library I use calibres Date column to record when I last did something significant to a 'book'.

BR

theducks · 06-18-2016, 07:42 PM

Clarify:
RAM is Memory
Disk is Storage

Calibre ALWAYS makes a copy of the File, which is why I suggested off loading (archive?) the original SOURCE FILE
Kovid (the creator) point out that during the ADD, there exist a (3rd) temporary copy that goes away when the ADD session posts to the DB. This is the reason for doing in small batches if you are drive space limited.
Calibre can use any LOCAL attached drive (no cloud or Shares).
Calibre can manage any quantity of Libraries. The only limit is the number that appears on the Quick Switch list. They still exist

kingseesar · 06-18-2016, 09:10 PM

Again, lots of super useful info and I must admit, that you've made this Ph.D. holder feel quite ignorant and inadequate in regards to understanding Calibre. I blatantly didn't understand the depth of capabilities and functionality. I was thinking it was just a reader that afforded some semblance of organization. Okay... Sounds like I need to start over with it and see if I can do more organizing and mapping out how I can simultaneously store the files and arrange them in libraries according to the size of the file collections.

Thanks so much to everyone! Cheers! I'll post in a while (read as a few weeks, probably) and let you know if it's working out for me. It'll take me a while to figure all this out and clear out what I have in Calibre now, then juggle the files and libraries and all that. I'll get it done eventually, though. I need to get more organized, obviously. Sorry if I seem to have wasted anyone's time, but I do greatly appreciate the responses and it's explained a lot! Thank you so much! More soon!
Will

theducks · 06-18-2016, 09:55 PM

We all have reasons for separate Libraries, but consider KISS

If the (base) Library has the Same Structure (columns and field values), consider 1 Library for all those... And Use Virtual Libraries (a filtered sub set) for your General usage. No which Library did I put that in

Reasons FOR: Isolation (Not PG-13

, Not to be viewed/used by shares...) ,Intake: Pre-process before moving to the Production Libraries,
Column (field) differences. Many extra columns, means performance hits. Optimized Library structure may be a better way.

Remember, you may only have 1 Library open (read-write, the GUI ) at a time. You can HAVE Multiple Libraries available (read only) Via the CLI Content (separate configured) servers

06-18-2016, 01:05 PM	#1
kingseesar Junior Member Posts: 6 Karma: 10 Join Date: Jun 2016 Device: Calibre	Help! Calibre is Eating All My Memory! Hi everyone! First off, apologies if this has been addressed before. I did several searches in the forum and haven't found my specific situation and desperately need some help: I just got Calibre two days ago after adding my epubs from old hard drives off another three computers. There's a TON of books, and I've been trying to add them to my library chunk or two at a time. I don't have much memory with which to start, to be fair, but I had about 19GB or so. Should be plenty for starters. So I started using the "Add Books" feature to organize my files. I hit the button and Calibre tells me it's going to take a while (read as the progress bar was moving very slowly and took almost 30 mins to get to 3%), so I went to bed, leaving my computer to finish importing books. When I woke up, I got a memory error message and found that my hard drive has no memory left at all! Calibre has used up 19GB of memory importing books! Is it duplicating them somewhere? Is it expanding metadata somewhere that's doubling or tripling the size of my files? What can I do to revert this nightmare? I thought Calibre was just going to make note of where my books were located on my computer and wouldn't be using any memory to add them to my library. I have about 2.5TB of books that I wanted to add, but if I can't do it in Calibre without doubling my memory use, there's no point in using it... Any help or suggestions will be greatly appreciated! Thanks in advance! Cheers! Will

06-18-2016, 04:38 PM	#9
BetterRed null operator (he/him) Posts: 21,646 Karma: 29710510 Join Date: Mar 2012 Location: Sydney Australia Device: none	@kingseesar - some people use calibre as a catalog only, and leave the book files where they are. Hopefully one of them will drop by and give some guidance. But here are some thoughts Firstly have a browse through this ==>> Calibre command line interface If you want calibre to extract metadata from the format files (EPUB, PDF etc) as it adds (i.e. you've checked Preferences->Add books->Read metadata from file contents...) then you will have to add the format files - but you do not have to keep them in the library. Once the book is added you can remove the format files - you should use calibre's Remove Formats feature to do this rather than your file manager. Using the file manager will result in the calibre library's database and its folder/file contents being inconsistent which is best avoided, Or, you could use the ebook-meta command to extract the metadata from the existing files into OPF files and add those to the calibre GUI. Calibre's going to retain an OPF anyway, they are used by its database restore feature. On Windows you could add a shortcut file to a book that references the current location. Alternatively you could put a hyperlink to the existing format file (folder) into the Comments column (or a similar custom column) or into an Arbitrary Identifier. The advantage of using shortcuts is that if you rename or move a shortcut target, Windows will try to repair any LNK files that refer to it - providing you haven't disabled the DLT Service. AFAIK there is no similar feature to repair broken file system URIs - on Windows, OSX or Linux. Using shortcut files or URLs (file:\\\blah de blah) precludes the use of some of the GUI features - particularly those relating to devices. However many features are also available from the command line - eg conversion and the aforementioned ebook-meta command. The calibredb command also has many useful features. One of the main attributes of calibre is its flexibility - i.e. there are many ways to skin a cat, other than dragging it backwards through a keyhole Added - have you taken the time to watch the Video Tutorials. I found them very useful when I started to use calibre - I'm someone who despises video 'instructions'. Watch a segment, then practise it on a test library, if it doesn't work as expected - re-wind/re-play/re-practise. BR Last edited by BetterRed; 06-18-2016 at 05:18 PM. Reason: See - Added

06-18-2016, 05:25 PM	#12
BetterRed null operator (he/him) Posts: 21,646 Karma: 29710510 Join Date: Mar 2012 Location: Sydney Australia Device: none	@kingseesar - The calibre viewer is a discrete program - ebook-viewer - you don't need the library manager to use it, same deal with the calibre editor, it's ebook-edit. I have them both in my OS file associations for EPUB - the viewer as the default. I keep separate libraries for books, comics, magazines/journals, picture albums and media (video and audio clips) - mainly because they have quite different metadata data elements, or I use the same metadata items differently - eg in one library I use calibres Date column to record when I last did something significant to a 'book'. BR Last edited by BetterRed; 06-18-2016 at 05:35 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Calibre throwing a Memory Exception	Thisiswhatiam	Calibre	6	06-17-2012 02:44 AM
Calibre takes up Too Much Memory	jerryprism	Calibre	14	04-23-2011 09:47 AM
PC Memory use of Calibre query	JustinD	Calibre	1	03-28-2011 07:29 PM
Classic Softroot 2.4.1 and Calibre 0.7.12 Memory Issue	6502programmer	Nook Developer's Corner	13	08-05-2010 02:16 PM

06-18-2016, 01:10 PM	#2
kingseesar Junior Member Posts: 6 Karma: 10 Join Date: Jun 2016 Device: Calibre	Slight update - When I clicked "OK" to exit the error message, 3.03GB of memory mysteriously returned to my hard drive. I still have no idea what's going on, either... Thanks again!

06-18-2016, 01:29 PM	#3
kovidgoyal creator of calibre Posts: 45,229 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Adding books requires approximately three times the disk space used by the actual book files during the adding process. SO either add in small batches or use an environment variable to move the calibre temp dir to a disk with more space.

06-18-2016, 02:16 PM	#5
kingseesar Junior Member Posts: 6 Karma: 10 Join Date: Jun 2016 Device: Calibre	Thanks Kovidgoyal and Jhowell! Cheer for getting back to me so fast! I'm rather surprised that Calibre doesn't just make note of where the books are and then gather metadata and open them from their current location. Sounds like I can't use Calibre, then, which is too bad, as I quite liked the interface and features, but I have so many books and comics on my hard drives and I certainly don't have the thousands of free gig space necessary to add the books to my Calibre library, which was the point of using it. And that's not an exaggeration. I have almost 2TB of comics alone, then there's the epubs I've been collecting for years, which is almost another 1TB, so Calibre just isn't feasible, sadly. Oh well... Thank you so much for your responses! I'll figure something out. Cheers! Will

06-18-2016, 03:03 PM	#6
DaltonST Deviser Posts: 2,265 Karma: 2090983 Join Date: Aug 2013 Location: Texas Device: none	@kingseesar: Calibre > Preferences > Add Books > Automatic Adding has the ability to automatically delete the source books being added after they have been added. DaltonST

06-18-2016, 03:35 PM	#8
kingseesar Junior Member Posts: 6 Karma: 10 Join Date: Jun 2016 Device: Calibre	@DaltonST : So... I have the option of deleting the original copies of the books once they have been added to Calibre, then? I'm not sure that works, either, only because that means I have to consolidate all my books onto one drive and, as before, I don't have that kind of space. I'd need a 3TB drive or bigger to do that, and I don't have that kind of money at the moment. Still, it's good to know that's a possibility. I've already added over 3000 books to Calibre's library, but I think they are all from the same drive where the library is stored. I assume there's a way to move the library to another drive, somehow? Maybe if I can get a new bigger drive, I can put them all there? I have four external drives and one internal drive, all with books/comics on them at the moment, thus I was hoping for something that would just make a note of where they are located and grab them as needed, but I'll keep this in mind for sure. Thank you! @theducks : and you are saying that the memory usage is temporary when adding books to the library. @jhowell above suggests that it duplicates the book in its entirety and even if Calibre does only use triple the memory temporarily, that only drops to double the memory used when adding a book. Is that correct? I'm happy to do small groups, if that helps, but it sounds like it won't in the long run. And again, I'd love to get a bigger drive, but that's out of my spending range at the moment. I've got enough space on my drives for now for daily use (around 400GB total between them all) and I do a lot of audio and video editing, so I need to keep that semi free, but once I get the funds, absolutely, I'll upgrade like a fiend! Thanks again to everyone for your responses! Cheers! This is all helpful. I certainly have a better understanding of how the program works and generates the library information; sadly, it's still not sounding like there's a feasible solution, yet, though, without a much, much, much bigger hard drive. I'll keep thinking on it and see what I can sort out. Thanks again! You all rock! \m/

06-18-2016, 05:15 PM	#11
kingseesar Junior Member Posts: 6 Karma: 10 Join Date: Jun 2016 Device: Calibre	@BetterRed : Okay. I think that makes sense. If I can get my head around that, then making a catalogue only would suffice, except I do need to be able to read the epubs using Calibre (or some reader). The other formats are not a problem. Cool! I'll look more into this. Cheer! @DaltonST : I thought that Calibre was duplicating my files somewhere of it's choice and I wasn't able to find in the preferences how to reassign the drive or directory into which Calibre puts the copied files, so I thought it was set, and thus the library had to be on the same drive as Calibre. obviously, I'm wrong, so I'll try and figure that out, too. As for separate libraries, potentially, as there are a few things I collect/read/research, but they will still be huge libraries. My Lovecraft-related collection is well over 5000 entires, and my comic collection is well over 25,000. The Smaller stuff would definitely work though (the Ethnomusicology (2000) and the Gothic Studies (900) and then the stragglers...) Nice! I'll see about splitting things up, then, if that works. Perhaps the problem is that I was just hoping to put everything in one place and not have to relocate anything, so I apologise if I'm just being lazy. Excellent! Thanks again everyone! Very helpful responses! Cheers! I really appreciate the assistance! Will

06-18-2016, 07:42 PM	#13
theducks Well trained by Cats Posts: 30,920 Karma: 60358908 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	Clarify: RAM is Memory Disk is Storage Calibre ALWAYS makes a copy of the File, which is why I suggested off loading (archive?) the original SOURCE FILE Kovid (the creator) point out that during the ADD, there exist a (3rd) temporary copy that goes away when the ADD session posts to the DB. This is the reason for doing in small batches if you are drive space limited. Calibre can use any LOCAL attached drive (no cloud or Shares). Calibre can manage any quantity of Libraries. The only limit is the number that appears on the Quick Switch list. They still exist

06-18-2016, 09:10 PM	#14
kingseesar Junior Member Posts: 6 Karma: 10 Join Date: Jun 2016 Device: Calibre	Again, lots of super useful info and I must admit, that you've made this Ph.D. holder feel quite ignorant and inadequate in regards to understanding Calibre. I blatantly didn't understand the depth of capabilities and functionality. I was thinking it was just a reader that afforded some semblance of organization. Okay... Sounds like I need to start over with it and see if I can do more organizing and mapping out how I can simultaneously store the files and arrange them in libraries according to the size of the file collections. Thanks so much to everyone! Cheers! I'll post in a while (read as a few weeks, probably) and let you know if it's working out for me. It'll take me a while to figure all this out and clear out what I have in Calibre now, then juggle the files and libraries and all that. I'll get it done eventually, though. I need to get more organized, obviously. Sorry if I seem to have wasted anyone's time, but I do greatly appreciate the responses and it's explained a lot! Thank you so much! More soon! Will

06-18-2016, 09:55 PM	#15
theducks Well trained by Cats Posts: 30,920 Karma: 60358908 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	We all have reasons for separate Libraries, but consider KISS If the (base) Library has the Same Structure (columns and field values), consider 1 Library for all those... And Use Virtual Libraries (a filtered sub set) for your General usage. No which Library did I put that in Reasons FOR: Isolation (Not PG-13 , Not to be viewed/used by shares...) ,Intake: Pre-process before moving to the Production Libraries, Column (field) differences. Many extra columns, means performance hits. Optimized Library structure may be a better way. Remember, you may only have 1 Library open (read-write, the GUI ) at a time. You can HAVE Multiple Libraries available (read only) Via the CLI Content (separate configured) servers

Advert

Advert