Register Guidelines E-Books Today's Posts Search

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 06-18-2016, 01:05 PM   #1
kingseesar
Junior Member
kingseesar began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Jun 2016
Device: Calibre
Unhappy Help! Calibre is Eating All My Memory!

Hi everyone!

First off, apologies if this has been addressed before. I did several searches in the forum and haven't found my specific situation and desperately need some help:

I just got Calibre two days ago after adding my epubs from old hard drives off another three computers. There's a TON of books, and I've been trying to add them to my library chunk or two at a time. I don't have much memory with which to start, to be fair, but I had about 19GB or so. Should be plenty for starters. So I started using the "Add Books" feature to organize my files. I hit the button and Calibre tells me it's going to take a while (read as the progress bar was moving very slowly and took almost 30 mins to get to 3%), so I went to bed, leaving my computer to finish importing books.

When I woke up, I got a memory error message and found that my hard drive has no memory left at all! Calibre has used up 19GB of memory importing books! Is it duplicating them somewhere? Is it expanding metadata somewhere that's doubling or tripling the size of my files? What can I do to revert this nightmare? I thought Calibre was just going to make note of where my books were located on my computer and wouldn't be using any memory to add them to my library. I have about 2.5TB of books that I wanted to add, but if I can't do it in Calibre without doubling my memory use, there's no point in using it...

Any help or suggestions will be greatly appreciated! Thanks in advance! Cheers!
Will
kingseesar is offline   Reply With Quote
Old 06-18-2016, 01:10 PM   #2
kingseesar
Junior Member
kingseesar began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Jun 2016
Device: Calibre
Slight update - When I clicked "OK" to exit the error message, 3.03GB of memory mysteriously returned to my hard drive. I still have no idea what's going on, either...

Thanks again!
kingseesar is offline   Reply With Quote
Advert
Old 06-18-2016, 01:29 PM   #3
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 45,229
Karma: 27110894
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Adding books requires approximately three times the disk space used by the actual book files during the adding process. SO either add in small batches or use an environment variable to move the calibre temp dir to a disk with more space.
kovidgoyal is offline   Reply With Quote
Old 06-18-2016, 01:43 PM   #4
jhowell
Grand Sorcerer
jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.jhowell ought to be getting tired of karma fortunes by now.
 
jhowell's Avatar
 
Posts: 7,046
Karma: 91577715
Join Date: Nov 2011
Location: Charlottesville, VA
Device: Kindles
Quote:
Originally Posted by kingseesar View Post
I thought Calibre was just going to make note of where my books were located on my computer and wouldn't be using any memory to add them to my library.
Calibre copies the actual book files into its own library directory structure. The originals will be left in place.
jhowell is online now   Reply With Quote
Old 06-18-2016, 02:16 PM   #5
kingseesar
Junior Member
kingseesar began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Jun 2016
Device: Calibre
Thanks Kovidgoyal and Jhowell! Cheer for getting back to me so fast!

I'm rather surprised that Calibre doesn't just make note of where the books are and then gather metadata and open them from their current location. Sounds like I can't use Calibre, then, which is too bad, as I quite liked the interface and features, but I have so many books and comics on my hard drives and I certainly don't have the thousands of free gig space necessary to add the books to my Calibre library, which was the point of using it. And that's not an exaggeration. I have almost 2TB of comics alone, then there's the epubs I've been collecting for years, which is almost another 1TB, so Calibre just isn't feasible, sadly.

Oh well... Thank you so much for your responses! I'll figure something out. Cheers!
Will
kingseesar is offline   Reply With Quote
Advert
Old 06-18-2016, 03:03 PM   #6
DaltonST
Deviser
DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.
 
DaltonST's Avatar
 
Posts: 2,265
Karma: 2090983
Join Date: Aug 2013
Location: Texas
Device: none
@kingseesar:

Calibre > Preferences > Add Books > Automatic Adding has the ability to automatically delete the source books being added after they have been added.


DaltonST
DaltonST is offline   Reply With Quote
Old 06-18-2016, 03:07 PM   #7
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 30,920
Karma: 60358908
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
Quote:
Originally Posted by kingseesar View Post
Thanks Kovidgoyal and Jhowell! Cheer for getting back to me so fast!

I'm rather surprised that Calibre doesn't just make note of where the books are and then gather metadata and open them from their current location. Sounds like I can't use Calibre, then, which is too bad, as I quite liked the interface and features, but I have so many books and comics on my hard drives and I certainly don't have the thousands of free gig space necessary to add the books to my Calibre library, which was the point of using it. And that's not an exaggeration. I have almost 2TB of comics alone, then there's the epubs I've been collecting for years, which is almost another 1TB, so Calibre just isn't feasible, sadly.

Oh well... Thank you so much for your responses! I'll figure something out. Cheers!
Will
Do a small group. Then offload those.
Repeat


(better yet: Get a bigger drive. Most sell with a clone the drive softwar package available. CloneZilla will do dual boot drives, where Acronis will not)
theducks is offline   Reply With Quote
Old 06-18-2016, 03:35 PM   #8
kingseesar
Junior Member
kingseesar began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Jun 2016
Device: Calibre
@DaltonST : So... I have the option of deleting the original copies of the books once they have been added to Calibre, then? I'm not sure that works, either, only because that means I have to consolidate all my books onto one drive and, as before, I don't have that kind of space. I'd need a 3TB drive or bigger to do that, and I don't have that kind of money at the moment. Still, it's good to know that's a possibility. I've already added over 3000 books to Calibre's library, but I think they are all from the same drive where the library is stored. I assume there's a way to move the library to another drive, somehow? Maybe if I can get a new bigger drive, I can put them all there? I have four external drives and one internal drive, all with books/comics on them at the moment, thus I was hoping for something that would just make a note of where they are located and grab them as needed, but I'll keep this in mind for sure. Thank you!

@theducks : and you are saying that the memory usage is temporary when adding books to the library. @jhowell above suggests that it duplicates the book in its entirety and even if Calibre does only use triple the memory temporarily, that only drops to double the memory used when adding a book. Is that correct? I'm happy to do small groups, if that helps, but it sounds like it won't in the long run.

And again, I'd love to get a bigger drive, but that's out of my spending range at the moment. I've got enough space on my drives for now for daily use (around 400GB total between them all) and I do a lot of audio and video editing, so I need to keep that semi free, but once I get the funds, absolutely, I'll upgrade like a fiend!

Thanks again to everyone for your responses! Cheers! This is all helpful. I certainly have a better understanding of how the program works and generates the library information; sadly, it's still not sounding like there's a feasible solution, yet, though, without a much, much, much bigger hard drive. I'll keep thinking on it and see what I can sort out. Thanks again! You all rock! \m/
kingseesar is offline   Reply With Quote
Old 06-18-2016, 04:38 PM   #9
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 21,646
Karma: 29710510
Join Date: Mar 2012
Location: Sydney Australia
Device: none
@kingseesar - some people use calibre as a catalog only, and leave the book files where they are. Hopefully one of them will drop by and give some guidance. But here are some thoughts

Firstly have a browse through this ==>> Calibre command line interface

If you want calibre to extract metadata from the format files (EPUB, PDF etc) as it adds (i.e. you've checked Preferences->Add books->Read metadata from file contents...) then you will have to add the format files - but you do not have to keep them in the library. Once the book is added you can remove the format files - you should use calibre's Remove Formats feature to do this rather than your file manager. Using the file manager will result in the calibre library's database and its folder/file contents being inconsistent which is best avoided,

Or, you could use the ebook-meta command to extract the metadata from the existing files into OPF files and add those to the calibre GUI. Calibre's going to retain an OPF anyway, they are used by its database restore feature.

On Windows you could add a shortcut file to a book that references the current location. Alternatively you could put a hyperlink to the existing format file (folder) into the Comments column (or a similar custom column) or into an Arbitrary Identifier. The advantage of using shortcuts is that if you rename or move a shortcut target, Windows will try to repair any LNK files that refer to it - providing you haven't disabled the DLT Service. AFAIK there is no similar feature to repair broken file system URIs - on Windows, OSX or Linux.

Using shortcut files or URLs (file:\\\blah de blah) precludes the use of some of the GUI features - particularly those relating to devices. However many features are also available from the command line - eg conversion and the aforementioned ebook-meta command. The calibredb command also has many useful features.

One of the main attributes of calibre is its flexibility - i.e. there are many ways to skin a cat, other than dragging it backwards through a keyhole

Added - have you taken the time to watch the Video Tutorials. I found them very useful when I started to use calibre - I'm someone who despises video 'instructions'. Watch a segment, then practise it on a test library, if it doesn't work as expected - re-wind/re-play/re-practise.

BR

Last edited by BetterRed; 06-18-2016 at 05:18 PM. Reason: See - Added
BetterRed is online now   Reply With Quote
Old 06-18-2016, 04:54 PM   #10
DaltonST
Deviser
DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.DaltonST ought to be getting tired of karma fortunes by now.
 
DaltonST's Avatar
 
Posts: 2,265
Karma: 2090983
Join Date: Aug 2013
Location: Texas
Device: none
Quote:
@DaltonST : So... I have the option of deleting the original copies of the books once they have been added to Calibre, then? I'm not sure that works, either, only because that means I have to consolidate all my books onto one drive and, as before, I don't have that kind of space.

Who said that you can have only a single Calibre library?

What makes you think that all Calibre libraries have to be on the same drive?

A different physical Calibre library based on broad differences might make sense. Comics do not need to be in the same physical library as Non-Fiction or as Fiction.


DaltonST
DaltonST is offline   Reply With Quote
Old 06-18-2016, 05:15 PM   #11
kingseesar
Junior Member
kingseesar began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Jun 2016
Device: Calibre
@BetterRed : Okay. I think that makes sense. If I can get my head around that, then making a catalogue only would suffice, except I do need to be able to read the epubs using Calibre (or some reader). The other formats are not a problem. Cool! I'll look more into this. Cheer!

@DaltonST : I thought that Calibre was duplicating my files somewhere of it's choice and I wasn't able to find in the preferences how to reassign the drive or directory into which Calibre puts the copied files, so I thought it was set, and thus the library had to be on the same drive as Calibre. obviously, I'm wrong, so I'll try and figure that out, too. As for separate libraries, potentially, as there are a few things I collect/read/research, but they will still be huge libraries. My Lovecraft-related collection is well over 5000 entires, and my comic collection is well over 25,000. The Smaller stuff would definitely work though (the Ethnomusicology (2000) and the Gothic Studies (900) and then the stragglers...) Nice! I'll see about splitting things up, then, if that works. Perhaps the problem is that I was just hoping to put everything in one place and not have to relocate anything, so I apologise if I'm just being lazy.

Excellent! Thanks again everyone! Very helpful responses! Cheers! I really appreciate the assistance!
Will
kingseesar is offline   Reply With Quote
Old 06-18-2016, 05:25 PM   #12
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 21,646
Karma: 29710510
Join Date: Mar 2012
Location: Sydney Australia
Device: none
@kingseesar - The calibre viewer is a discrete program - ebook-viewer - you don't need the library manager to use it, same deal with the calibre editor, it's ebook-edit. I have them both in my OS file associations for EPUB - the viewer as the default.

I keep separate libraries for books, comics, magazines/journals, picture albums and media (video and audio clips) - mainly because they have quite different metadata data elements, or I use the same metadata items differently - eg in one library I use calibres Date column to record when I last did something significant to a 'book'.

BR

Last edited by BetterRed; 06-18-2016 at 05:35 PM.
BetterRed is online now   Reply With Quote
Old 06-18-2016, 07:42 PM   #13
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 30,920
Karma: 60358908
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
Clarify:
RAM is Memory
Disk is Storage

Calibre ALWAYS makes a copy of the File, which is why I suggested off loading (archive?) the original SOURCE FILE
Kovid (the creator) point out that during the ADD, there exist a (3rd) temporary copy that goes away when the ADD session posts to the DB. This is the reason for doing in small batches if you are drive space limited.
Calibre can use any LOCAL attached drive (no cloud or Shares).
Calibre can manage any quantity of Libraries. The only limit is the number that appears on the Quick Switch list. They still exist
theducks is offline   Reply With Quote
Old 06-18-2016, 09:10 PM   #14
kingseesar
Junior Member
kingseesar began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Jun 2016
Device: Calibre
Again, lots of super useful info and I must admit, that you've made this Ph.D. holder feel quite ignorant and inadequate in regards to understanding Calibre. I blatantly didn't understand the depth of capabilities and functionality. I was thinking it was just a reader that afforded some semblance of organization. Okay... Sounds like I need to start over with it and see if I can do more organizing and mapping out how I can simultaneously store the files and arrange them in libraries according to the size of the file collections.

Thanks so much to everyone! Cheers! I'll post in a while (read as a few weeks, probably) and let you know if it's working out for me. It'll take me a while to figure all this out and clear out what I have in Calibre now, then juggle the files and libraries and all that. I'll get it done eventually, though. I need to get more organized, obviously. Sorry if I seem to have wasted anyone's time, but I do greatly appreciate the responses and it's explained a lot! Thank you so much! More soon!
Will
kingseesar is offline   Reply With Quote
Old 06-18-2016, 09:55 PM   #15
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 30,920
Karma: 60358908
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
We all have reasons for separate Libraries, but consider KISS

If the (base) Library has the Same Structure (columns and field values), consider 1 Library for all those... And Use Virtual Libraries (a filtered sub set) for your General usage. No which Library did I put that in

Reasons FOR: Isolation (Not PG-13 , Not to be viewed/used by shares...) ,Intake: Pre-process before moving to the Production Libraries,
Column (field) differences. Many extra columns, means performance hits. Optimized Library structure may be a better way.

Remember, you may only have 1 Library open (read-write, the GUI ) at a time. You can HAVE Multiple Libraries available (read only) Via the CLI Content (separate configured) servers
theducks is offline   Reply With Quote
Reply

Tags
adding books calibre, calibre memory error


Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Calibre throwing a Memory Exception Thisiswhatiam Calibre 6 06-17-2012 02:44 AM
Calibre takes up Too Much Memory jerryprism Calibre 14 04-23-2011 09:47 AM
PC Memory use of Calibre query JustinD Calibre 1 03-28-2011 07:29 PM
Classic Softroot 2.4.1 and Calibre 0.7.12 Memory Issue 6502programmer Nook Developer's Corner 13 08-05-2010 02:16 PM


All times are GMT -4. The time now is 07:08 AM.


MobileRead.com is a privately owned, operated and funded community.