11-11-2016, 06:40 PM | #1 |
Fanatic
Posts: 515
Karma: 1470724
Join Date: Jul 2013
Location: Quebec CA
Device: android 4 (samsung tablet and asus tablet)
|
A few questions about how size of library impacts memory usage
I'm trying to see how much the size of the library affects the memory usage.
My larger library is currently using somewhere between 1.7gig and 2.4gig of ram. It was seeing the 2.4 gig of ram in use when Calibre was just sitting open that made me curious about what determines how much memory is used. I had been doing maintenance earlier so there was currently no editing, additions, conversions being done. The net sharing was setup so my housemates could access the net sharing if they wanted but no one was accessing it (does the net sharing use a lot of ram?). This made me kind of curious about much ram calibre used. Also, is it the size on the disk or the number of entries? Will reducing the number of custom columns significantly reduce the memory usage in a 200,000 title library? if most of a custom column is blank is it still using memory if this is the case? I did look in the help manual but didn't see anything about memory usage. |
11-11-2016, 08:17 PM | #2 |
null operator (he/him)
Posts: 20,538
Karma: 26944418
Join Date: Mar 2012
Location: Sydney Australia
Device: none
|
@LadyKate - 2.4 gig (i.e. 2,400 Megabytes) seems like a really massive number to me Even memory hogs like Firefox, Autocad, and my video editors don't get that high - collectively. How much time does calibre take to start and switch between those two libraries?
Anyway pending Kovid's definitive answers, here are some thoughts. As I understand matters, when calibre is running, the database is memory resident. It gets loaded at start and on a library switch. So I imagine the number of books, the number of custom columns, and the size of the column data items would be major determinants of the amount of memory needed for the in-memory database. I think most data values are normalised - i.e. there will only be one instance of the string 'Miguel de Cervantes Saavedra'. Assuming they are held in memory, and I don't know that they're not, if you have lots of very large Comments or custom long text columns, I imagine that could have a major impact on memory usage. I normally run three library servers via command files. The main usage is to access metadata, I rarely access format files from the server. I've not looked to see how much memory they use, but they've never given me any reason for concern - most of the time I assume they'd be paged out. BR |
11-11-2016, 08:50 PM | #3 |
creator of calibre
Posts: 43,830
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
That's insane. What do you have in your databases? The entire text of books? Memory usage is approximately:
constant1 + constant2 * size of metadata.db because the entire metadata.db is laoded into RAM (but unpacked and stored in an object layout for speed). Custom columns behave no differently from builtin ones. For every column that you have the required memory goes up by size needed for an entry in the column * number of books in the library (technically it is not quite that large, because empty items use less memory for most column types). Both constant1 and constant2 are only approximately constants. I've made them constant only to simplify the discussion. On my machine with a library with 50,000 books, calibre uses 600MB. With a library of 1000 books it uses 100MB. The content server does not use any significant amounts of memory, but that is easy to test, simply turn it off and compare memory with it on and with it off. |
11-12-2016, 03:02 AM | #4 | |
Fanatic
Posts: 515
Karma: 1470724
Join Date: Jul 2013
Location: Quebec CA
Device: android 4 (samsung tablet and asus tablet)
|
Quote:
The comments are filled in as I search for ID links and I usually use Goodreads if it is available. Not sure if it matters but the custom columns are 4 y/n/blank, 1 date, 1 built from other columns and 7 text columns. It makes no real difference if the content server is on or off for loading time or memory usage. After restarting Calibre several times and checking what preferences and columns were set, for some reason the time to enter Calibre is under a minute (25 seconds). Earlier today it took much longer to load. My system at this time has two text editors with a total of 6 ascii text windows open. Chrome with 9 tabs (that's kind of light, usually have about 22 or so tabs open) and Firefox with 6 tabs. ummm I also have mirc and ultrafilesearchlite running and of course task manager. Oh yes, I also have three explorer windows open and one calibre ebook viewer open. The version of Calibre that I have installed is 2.66 (I only download when there is a significant change or I have a problem) but I didn't see anything about memory management in the update notes when I looked at them. Calibre had been open for about three days without being closed when I noticed it was using 2400meg. That seemed to be a bit much. I had been downloading metadata and doing conversions and adding tags and saving selected files to disk. Oh yes, I had also been running duplicate file search and checking each of the files to see which to keep. Some of my custom columns are nearly empty but... I may create a duplicate of this library and see how much difference in memory usage there is as I start delete the custom columns. By the way, I was not complaining about the memory usage. Just started looking at it as I can't afford a laptop with near as much ram as my desktop system and wanted to see just what my normal usage was to see how tight things would be. |
|
11-12-2016, 05:05 AM | #5 |
creator of calibre
Posts: 43,830
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Most calibre tasks run in separate short-lived worker processes, so they cannot contribute to long term memory usage of calibre. If you find that memory usage is growing over time, then it is likely something is leaking memory. The best way to find out what is to perform some calibre operations that you suspect repeatedly and watch the meory usage. Note that some increase is expected because python the language calibre is written in often does not immediately release memory back to the OS, as a performance optimization. But if you perform the same operation five times and get a roughly equal increase in mem usage each time, then that is likely a leak.
|
11-14-2016, 09:59 AM | #6 |
Wizard
Posts: 2,016
Karma: 13471689
Join Date: Oct 2007
Location: Almere, The Netherlands
Device: Kobo Sage
|
My metadata.db is 455MB (almost 200K books), and a freshly started 64-bit Calibre 2.71 takes 1.8GB of RAM.
Just another data point. |
11-14-2016, 08:07 PM | #7 | |
Fanatic
Posts: 515
Karma: 1470724
Join Date: Jul 2013
Location: Quebec CA
Device: android 4 (samsung tablet and asus tablet)
|
Quote:
I was just kind of curious. Thank you for your input mobovenka. kovidgoyal - I have not noticed anything as being a real memory leak. It does appear that when I do a duplicate search the result increases memory usage. That would make sense to me. |
|
11-14-2016, 10:05 PM | #8 |
creator of calibre
Posts: 43,830
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
You are doing a duplicate search using the duplicate finder plugin?
|
11-15-2016, 03:26 AM | #9 | |
Wizard
Posts: 2,016
Karma: 13471689
Join Date: Oct 2007
Location: Almere, The Netherlands
Device: Kobo Sage
|
Quote:
You mention duplicate searches; Like Kovid, I'm assuming you use the Find Duplicates plugin for that. I have found an issue with it; If I try a duplicate search with 'Soundex' matching for the Title, it starts eating memory until it either runs into the 32-bit process limit and blows up, or starts swapping my PC to death (with the 64-bit version). All other matching types are OK. It's not what you are seeing, obviously, but it may be related in some way. |
|
11-18-2016, 04:54 AM | #10 | |
Fanatic
Posts: 515
Karma: 1470724
Join Date: Jul 2013
Location: Quebec CA
Device: android 4 (samsung tablet and asus tablet)
|
Quote:
Won't be for a while though. Headaches make it difficult to concentrate. (horrid sinus headaches...different and not quite as bad as migraines but still painful) |
|
Thread Tools | Search this Thread |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Calibre memory usage - Linux | Dapper | Calibre | 11 | 03-07-2016 11:03 PM |
n2a card memory usage | jvik | Android Devices | 1 | 04-10-2015 02:28 PM |
Memory usage | jessie102 | Calibre | 4 | 01-29-2015 12:44 PM |
Extreme memory usage | MJBoa | Calibre | 1 | 02-12-2010 11:37 AM |
Memory Usage | BCCISProf | Sony Reader | 15 | 05-16-2007 08:11 PM |