![]() |
#1 |
Junior Member
![]() Posts: 6
Karma: 10
Join Date: Jun 2016
Device: Calibre
|
![]()
Hi everyone!
First off, apologies if this has been addressed before. I did several searches in the forum and haven't found my specific situation and desperately need some help: I just got Calibre two days ago after adding my epubs from old hard drives off another three computers. There's a TON of books, and I've been trying to add them to my library chunk or two at a time. I don't have much memory with which to start, to be fair, but I had about 19GB or so. Should be plenty for starters. So I started using the "Add Books" feature to organize my files. I hit the button and Calibre tells me it's going to take a while (read as the progress bar was moving very slowly and took almost 30 mins to get to 3%), so I went to bed, leaving my computer to finish importing books. When I woke up, I got a memory error message and found that my hard drive has no memory left at all! Calibre has used up 19GB of memory importing books! Is it duplicating them somewhere? Is it expanding metadata somewhere that's doubling or tripling the size of my files? What can I do to revert this nightmare? I thought Calibre was just going to make note of where my books were located on my computer and wouldn't be using any memory to add them to my library. I have about 2.5TB of books that I wanted to add, but if I can't do it in Calibre without doubling my memory use, there's no point in using it... Any help or suggestions will be greatly appreciated! Thanks in advance! Cheers! Will ![]() |
![]() |
![]() |
![]() |
#2 |
Junior Member
![]() Posts: 6
Karma: 10
Join Date: Jun 2016
Device: Calibre
|
Slight update - When I clicked "OK" to exit the error message, 3.03GB of memory mysteriously returned to my hard drive. I still have no idea what's going on, either...
Thanks again! |
![]() |
![]() |
Advert | |
|
![]() |
#3 |
creator of calibre
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 45,229
Karma: 27110894
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Adding books requires approximately three times the disk space used by the actual book files during the adding process. SO either add in small batches or use an environment variable to move the calibre temp dir to a disk with more space.
|
![]() |
![]() |
![]() |
#4 |
Grand Sorcerer
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 7,046
Karma: 91577715
Join Date: Nov 2011
Location: Charlottesville, VA
Device: Kindles
|
Calibre copies the actual book files into its own library directory structure. The originals will be left in place.
|
![]() |
![]() |
![]() |
#5 |
Junior Member
![]() Posts: 6
Karma: 10
Join Date: Jun 2016
Device: Calibre
|
Thanks Kovidgoyal and Jhowell! Cheer for getting back to me so fast!
I'm rather surprised that Calibre doesn't just make note of where the books are and then gather metadata and open them from their current location. Sounds like I can't use Calibre, then, which is too bad, as I quite liked the interface and features, but I have so many books and comics on my hard drives and I certainly don't have the thousands of free gig space necessary to add the books to my Calibre library, which was the point of using it. And that's not an exaggeration. I have almost 2TB of comics alone, then there's the epubs I've been collecting for years, which is almost another 1TB, so Calibre just isn't feasible, sadly. Oh well... Thank you so much for your responses! I'll figure something out. Cheers! Will |
![]() |
![]() |
Advert | |
|
![]() |
#6 |
Deviser
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 2,265
Karma: 2090983
Join Date: Aug 2013
Location: Texas
Device: none
|
@kingseesar:
Calibre > Preferences > Add Books > Automatic Adding has the ability to automatically delete the source books being added after they have been added. DaltonST |
![]() |
![]() |
![]() |
#7 | |
Well trained by Cats
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 30,920
Karma: 60358908
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
|
Quote:
Repeat ![]() (better yet: Get a bigger drive. Most sell with a clone the drive softwar package available. CloneZilla will do dual boot drives, where Acronis will not) |
|
![]() |
![]() |
![]() |
#8 |
Junior Member
![]() Posts: 6
Karma: 10
Join Date: Jun 2016
Device: Calibre
|
@DaltonST : So... I have the option of deleting the original copies of the books once they have been added to Calibre, then? I'm not sure that works, either, only because that means I have to consolidate all my books onto one drive and, as before, I don't have that kind of space. I'd need a 3TB drive or bigger to do that, and I don't have that kind of money at the moment. Still, it's good to know that's a possibility. I've already added over 3000 books to Calibre's library, but I think they are all from the same drive where the library is stored. I assume there's a way to move the library to another drive, somehow? Maybe if I can get a new bigger drive, I can put them all there? I have four external drives and one internal drive, all with books/comics on them at the moment, thus I was hoping for something that would just make a note of where they are located and grab them as needed, but I'll keep this in mind for sure. Thank you!
@theducks : and you are saying that the memory usage is temporary when adding books to the library. @jhowell above suggests that it duplicates the book in its entirety and even if Calibre does only use triple the memory temporarily, that only drops to double the memory used when adding a book. Is that correct? I'm happy to do small groups, if that helps, but it sounds like it won't in the long run. And again, I'd love to get a bigger drive, but that's out of my spending range at the moment. I've got enough space on my drives for now for daily use (around 400GB total between them all) and I do a lot of audio and video editing, so I need to keep that semi free, but once I get the funds, absolutely, I'll upgrade like a fiend! Thanks again to everyone for your responses! Cheers! This is all helpful. I certainly have a better understanding of how the program works and generates the library information; sadly, it's still not sounding like there's a feasible solution, yet, though, without a much, much, much bigger hard drive. I'll keep thinking on it and see what I can sort out. Thanks again! You all rock! \m/ ![]() |
![]() |
![]() |
![]() |
#9 |
null operator (he/him)
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 21,646
Karma: 29710510
Join Date: Mar 2012
Location: Sydney Australia
Device: none
|
@kingseesar - some people use calibre as a catalog only, and leave the book files where they are. Hopefully one of them will drop by and give some guidance. But here are some thoughts
Firstly have a browse through this ==>> Calibre command line interface If you want calibre to extract metadata from the format files (EPUB, PDF etc) as it adds (i.e. you've checked Preferences->Add books->Read metadata from file contents...) then you will have to add the format files - but you do not have to keep them in the library. Once the book is added you can remove the format files - you should use calibre's Remove Formats feature to do this rather than your file manager. Using the file manager will result in the calibre library's database and its folder/file contents being inconsistent which is best avoided, Or, you could use the ebook-meta command to extract the metadata from the existing files into OPF files and add those to the calibre GUI. Calibre's going to retain an OPF anyway, they are used by its database restore feature. On Windows you could add a shortcut file to a book that references the current location. Alternatively you could put a hyperlink to the existing format file (folder) into the Comments column (or a similar custom column) or into an Arbitrary Identifier. The advantage of using shortcuts is that if you rename or move a shortcut target, Windows will try to repair any LNK files that refer to it - providing you haven't disabled the DLT Service. AFAIK there is no similar feature to repair broken file system URIs - on Windows, OSX or Linux. Using shortcut files or URLs (file:\\\blah de blah) precludes the use of some of the GUI features - particularly those relating to devices. However many features are also available from the command line - eg conversion and the aforementioned ebook-meta command. The calibredb command also has many useful features. One of the main attributes of calibre is its flexibility - i.e. there are many ways to skin a cat, other than dragging it backwards through a keyhole Added - have you taken the time to watch the Video Tutorials. I found them very useful when I started to use calibre - I'm someone who despises video 'instructions'. Watch a segment, then practise it on a test library, if it doesn't work as expected - re-wind/re-play/re-practise. BR Last edited by BetterRed; 06-18-2016 at 05:18 PM. Reason: See - Added |
![]() |
![]() |
![]() |
#10 | |
Deviser
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 2,265
Karma: 2090983
Join Date: Aug 2013
Location: Texas
Device: none
|
Quote:
Who said that you can have only a single Calibre library? What makes you think that all Calibre libraries have to be on the same drive? A different physical Calibre library based on broad differences might make sense. Comics do not need to be in the same physical library as Non-Fiction or as Fiction. DaltonST |
|
![]() |
![]() |
![]() |
#11 |
Junior Member
![]() Posts: 6
Karma: 10
Join Date: Jun 2016
Device: Calibre
|
@BetterRed : Okay. I think that makes sense. If I can get my head around that, then making a catalogue only would suffice, except I do need to be able to read the epubs using Calibre (or some reader). The other formats are not a problem. Cool! I'll look more into this. Cheer!
@DaltonST : I thought that Calibre was duplicating my files somewhere of it's choice and I wasn't able to find in the preferences how to reassign the drive or directory into which Calibre puts the copied files, so I thought it was set, and thus the library had to be on the same drive as Calibre. obviously, I'm wrong, so I'll try and figure that out, too. As for separate libraries, potentially, as there are a few things I collect/read/research, but they will still be huge libraries. My Lovecraft-related collection is well over 5000 entires, and my comic collection is well over 25,000. The Smaller stuff would definitely work though (the Ethnomusicology (2000) and the Gothic Studies (900) and then the stragglers...) Nice! I'll see about splitting things up, then, if that works. Perhaps the problem is that I was just hoping to put everything in one place and not have to relocate anything, so I apologise if I'm just being lazy. Excellent! Thanks again everyone! Very helpful responses! Cheers! I really appreciate the assistance! Will |
![]() |
![]() |
![]() |
#12 |
null operator (he/him)
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 21,646
Karma: 29710510
Join Date: Mar 2012
Location: Sydney Australia
Device: none
|
@kingseesar - The calibre viewer is a discrete program - ebook-viewer - you don't need the library manager to use it, same deal with the calibre editor, it's ebook-edit. I have them both in my OS file associations for EPUB - the viewer as the default.
I keep separate libraries for books, comics, magazines/journals, picture albums and media (video and audio clips) - mainly because they have quite different metadata data elements, or I use the same metadata items differently - eg in one library I use calibres Date column to record when I last did something significant to a 'book'. BR Last edited by BetterRed; 06-18-2016 at 05:35 PM. |
![]() |
![]() |
![]() |
#13 |
Well trained by Cats
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 30,920
Karma: 60358908
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
|
Clarify:
RAM is Memory Disk is Storage Calibre ALWAYS makes a copy of the File, which is why I suggested off loading (archive?) the original SOURCE FILE Kovid (the creator) point out that during the ADD, there exist a (3rd) temporary copy that goes away when the ADD session posts to the DB. This is the reason for doing in small batches if you are drive space limited. Calibre can use any LOCAL attached drive (no cloud or Shares). Calibre can manage any quantity of Libraries. The only limit is the number that appears on the Quick Switch list. They still exist |
![]() |
![]() |
![]() |
#14 |
Junior Member
![]() Posts: 6
Karma: 10
Join Date: Jun 2016
Device: Calibre
|
Again, lots of super useful info and I must admit, that you've made this Ph.D. holder feel quite ignorant and inadequate in regards to understanding Calibre. I blatantly didn't understand the depth of capabilities and functionality. I was thinking it was just a reader that afforded some semblance of organization. Okay... Sounds like I need to start over with it and see if I can do more organizing and mapping out how I can simultaneously store the files and arrange them in libraries according to the size of the file collections.
Thanks so much to everyone! Cheers! I'll post in a while (read as a few weeks, probably) and let you know if it's working out for me. It'll take me a while to figure all this out and clear out what I have in Calibre now, then juggle the files and libraries and all that. I'll get it done eventually, though. I need to get more organized, obviously. Sorry if I seem to have wasted anyone's time, but I do greatly appreciate the responses and it's explained a lot! Thank you so much! More soon! Will |
![]() |
![]() |
![]() |
#15 |
Well trained by Cats
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 30,920
Karma: 60358908
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
|
We all have reasons for separate Libraries, but consider KISS
![]() If the (base) Library has the Same Structure (columns and field values), consider 1 Library for all those... And Use Virtual Libraries (a filtered sub set) for your General usage. No which Library did I put that in ![]() ![]() Reasons FOR: Isolation (Not PG-13 ![]() Column (field) differences. Many extra columns, means performance hits. Optimized Library structure may be a better way. Remember, you may only have 1 Library open (read-write, the GUI ) at a time. You can HAVE Multiple Libraries available (read only) Via the CLI Content (separate configured) servers |
![]() |
![]() |
![]() |
Tags |
adding books calibre, calibre memory error |
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
Calibre throwing a Memory Exception | Thisiswhatiam | Calibre | 6 | 06-17-2012 02:44 AM |
Calibre takes up Too Much Memory | jerryprism | Calibre | 14 | 04-23-2011 09:47 AM |
PC Memory use of Calibre query | JustinD | Calibre | 1 | 03-28-2011 07:29 PM |
Classic Softroot 2.4.1 and Calibre 0.7.12 Memory Issue | 6502programmer | Nook Developer's Corner | 13 | 08-05-2010 02:16 PM |