Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 11-11-2016, 06:40 PM   #1
LadyKate
Fanatic
LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.
 
Posts: 511
Karma: 1470724
Join Date: Jul 2013
Location: Quebec CA
Device: android 4 (samsung tablet and asus tablet)
A few questions about how size of library impacts memory usage

I'm trying to see how much the size of the library affects the memory usage.

My larger library is currently using somewhere between 1.7gig and 2.4gig of ram. It was seeing the 2.4 gig of ram in use when Calibre was just sitting open that made me curious about what determines how much memory is used. I had been doing maintenance earlier so there was currently no editing, additions, conversions being done. The net sharing was setup so my housemates could access the net sharing if they wanted but no one was accessing it (does the net sharing use a lot of ram?).

This made me kind of curious about much ram calibre used.
Also, is it the size on the disk or the number of entries?

Will reducing the number of custom columns significantly reduce the memory usage in a 200,000 title library? if most of a custom column is blank is it still using memory if this is the case?

I did look in the help manual but didn't see anything about memory usage.
LadyKate is offline   Reply With Quote
Old 11-11-2016, 08:17 PM   #2
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 16,328
Karma: 19147306
Join Date: Mar 2012
Location: Sydney Australia
Device: none
@LadyKate - 2.4 gig (i.e. 2,400 Megabytes) seems like a really massive number to me Even memory hogs like Firefox, Autocad, and my video editors don't get that high - collectively. How much time does calibre take to start and switch between those two libraries?

Anyway pending Kovid's definitive answers, here are some thoughts.

As I understand matters, when calibre is running, the database is memory resident. It gets loaded at start and on a library switch. So I imagine the number of books, the number of custom columns, and the size of the column data items would be major determinants of the amount of memory needed for the in-memory database. I think most data values are normalised - i.e. there will only be one instance of the string 'Miguel de Cervantes Saavedra'.

Assuming they are held in memory, and I don't know that they're not, if you have lots of very large Comments or custom long text columns, I imagine that could have a major impact on memory usage.

I normally run three library servers via command files. The main usage is to access metadata, I rarely access format files from the server. I've not looked to see how much memory they use, but they've never given me any reason for concern - most of the time I assume they'd be paged out.

BR
BetterRed is offline   Reply With Quote
Advert
Old 11-11-2016, 08:50 PM   #3
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 37,324
Karma: 16448187
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
That's insane. What do you have in your databases? The entire text of books? Memory usage is approximately:

constant1 + constant2 * size of metadata.db

because the entire metadata.db is laoded into RAM (but unpacked and stored in an object layout for speed). Custom columns behave no differently from builtin ones. For every column that you have the required memory goes up by size needed for an entry in the column * number of books in the library (technically it is not quite that large, because empty items use less memory for most column types). Both constant1 and constant2 are only approximately constants. I've made them constant only to simplify the discussion.

On my machine with a library with 50,000 books, calibre uses 600MB. With a library of 1000 books it uses 100MB.

The content server does not use any significant amounts of memory, but that is easy to test, simply turn it off and compare memory with it on and with it off.
kovidgoyal is online now   Reply With Quote
Old 11-12-2016, 03:02 AM   #4
LadyKate
Fanatic
LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.
 
Posts: 511
Karma: 1470724
Join Date: Jul 2013
Location: Quebec CA
Device: android 4 (samsung tablet and asus tablet)
Quote:
Originally Posted by kovidgoyal View Post
That's insane. What do you have in your databases? The entire text of books? Memory usage is approximately:

constant1 + constant2 * size of metadata.db

because the entire metadata.db is laoded into RAM (but unpacked and stored in an object layout for speed). Custom columns behave no differently from builtin ones. For every column that you have the required memory goes up by size needed for an entry in the column * number of books in the library (technically it is not quite that large, because empty items use less memory for most column types). Both constant1 and constant2 are only approximately constants. I've made them constant only to simplify the discussion.

On my machine with a library with 50,000 books, calibre uses 600MB. With a library of 1000 books it uses 100MB.

The content server does not use any significant amounts of memory, but that is easy to test, simply turn it off and compare memory with it on and with it off.
Ok That I understand. My metadata.db is at 418695. There are around 240,000 books. I have 12, soon to be 13 custom columns. I replace tags with my own to help me determine what I want to read next...

The comments are filled in as I search for ID links and I usually use Goodreads if it is available.

Not sure if it matters but the custom columns are 4 y/n/blank, 1 date, 1 built from other columns and 7 text columns.

It makes no real difference if the content server is on or off for loading time or memory usage.

After restarting Calibre several times and checking what preferences and columns were set, for some reason the time to enter Calibre is under a minute (25 seconds). Earlier today it took much longer to load. My system at this time has two text editors with a total of 6 ascii text windows open. Chrome with 9 tabs (that's kind of light, usually have about 22 or so tabs open) and Firefox with 6 tabs. ummm I also have mirc and ultrafilesearchlite running and of course task manager. Oh yes, I also have three explorer windows open and one calibre ebook viewer open.

The version of Calibre that I have installed is 2.66 (I only download when there is a significant change or I have a problem) but I didn't see anything about memory management in the update notes when I looked at them.

Calibre had been open for about three days without being closed when I noticed it was using 2400meg. That seemed to be a bit much. I had been downloading metadata and doing conversions and adding tags and saving selected files to disk. Oh yes, I had also been running duplicate file search and checking each of the files to see which to keep.

Some of my custom columns are nearly empty but... I may create a duplicate of this library and see how much difference in memory usage there is as I start delete the custom columns.

By the way, I was not complaining about the memory usage. Just started looking at it as I can't afford a laptop with near as much ram as my desktop system and wanted to see just what my normal usage was to see how tight things would be.
LadyKate is offline   Reply With Quote
Old 11-12-2016, 05:05 AM   #5
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 37,324
Karma: 16448187
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Most calibre tasks run in separate short-lived worker processes, so they cannot contribute to long term memory usage of calibre. If you find that memory usage is growing over time, then it is likely something is leaking memory. The best way to find out what is to perform some calibre operations that you suspect repeatedly and watch the meory usage. Note that some increase is expected because python the language calibre is written in often does not immediately release memory back to the OS, as a performance optimization. But if you perform the same operation five times and get a roughly equal increase in mem usage each time, then that is likely a leak.
kovidgoyal is online now   Reply With Quote
Advert
Old 11-14-2016, 09:59 AM   #6
mbovenka
Wizard
mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.
 
Posts: 1,842
Karma: 6604015
Join Date: Oct 2007
Location: Almere, The Netherlands
Device: Kobo Forma
My metadata.db is 455MB (almost 200K books), and a freshly started 64-bit Calibre 2.71 takes 1.8GB of RAM.

Just another data point.
mbovenka is offline   Reply With Quote
Old 11-14-2016, 08:07 PM   #7
LadyKate
Fanatic
LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.
 
Posts: 511
Karma: 1470724
Join Date: Jul 2013
Location: Quebec CA
Device: android 4 (samsung tablet and asus tablet)
Quote:
Originally Posted by mbovenka View Post
My metadata.db is 455MB (almost 200K books), and a freshly started 64-bit Calibre 2.71 takes 1.8GB of RAM.

Just another data point.
So it sounds like the range of 1.7 to 2.4 is normal considering the size of my library. It seems to be dependent on how many duplicate searches etc. that I do and perhaps how long Calibre has been running without restarting.

I was just kind of curious. Thank you for your input mobovenka.

kovidgoyal - I have not noticed anything as being a real memory leak. It does appear that when I do a duplicate search the result increases memory usage. That would make sense to me.
LadyKate is offline   Reply With Quote
Old 11-14-2016, 10:05 PM   #8
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 37,324
Karma: 16448187
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
You are doing a duplicate search using the duplicate finder plugin?
kovidgoyal is online now   Reply With Quote
Old 11-15-2016, 03:26 AM   #9
mbovenka
Wizard
mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.mbovenka ought to be getting tired of karma fortunes by now.
 
Posts: 1,842
Karma: 6604015
Join Date: Oct 2007
Location: Almere, The Netherlands
Device: Kobo Forma
Quote:
Originally Posted by LadyKate View Post
So it sounds like the range of 1.7 to 2.4 is normal considering the size of my library. It seems to be dependent on how many duplicate searches etc. that I do and perhaps how long Calibre has been running without restarting.

I was just kind of curious. Thank you for your input mobovenka.

kovidgoyal - I have not noticed anything as being a real memory leak. It does appear that when I do a duplicate search the result increases memory usage. That would make sense to me.
Yes, that seems normal given that your library is about 20% larger than mine, with memory usage going up linearly with number of books (assuming 'constant1' can be considered small at such library sizes).

You mention duplicate searches; Like Kovid, I'm assuming you use the Find Duplicates plugin for that. I have found an issue with it; If I try a duplicate search with 'Soundex' matching for the Title, it starts eating memory until it either runs into the 32-bit process limit and blows up, or starts swapping my PC to death (with the 64-bit version).

All other matching types are OK.

It's not what you are seeing, obviously, but it may be related in some way.
mbovenka is offline   Reply With Quote
Old 11-18-2016, 04:54 AM   #10
LadyKate
Fanatic
LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.LadyKate ought to be getting tired of karma fortunes by now.
 
Posts: 511
Karma: 1470724
Join Date: Jul 2013
Location: Quebec CA
Device: android 4 (samsung tablet and asus tablet)
Quote:
Originally Posted by mbovenka View Post
Yes, that seems normal given that your library is about 20% larger than mine, with memory usage going up linearly with number of books (assuming 'constant1' can be considered small at such library sizes).

You mention duplicate searches; Like Kovid, I'm assuming you use the Find Duplicates plugin for that. I have found an issue with it; If I try a duplicate search with 'Soundex' matching for the Title, it starts eating memory until it either runs into the 32-bit process limit and blows up, or starts swapping my PC to death (with the 64-bit version).

All other matching types are OK.

It's not what you are seeing, obviously, but it may be related in some way.
I'm thinking I'll play with a smaller library (after starting Calibre in the small test library) and play with it.

Won't be for a while though. Headaches make it difficult to concentrate. (horrid sinus headaches...different and not quite as bad as migraines but still painful)
LadyKate is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Calibre memory usage - Linux Dapper Calibre 11 03-07-2016 11:03 PM
n2a card memory usage jvik Android Devices 1 04-10-2015 02:28 PM
Memory usage jessie102 Calibre 4 01-29-2015 12:44 PM
Extreme memory usage MJBoa Calibre 1 02-12-2010 11:37 AM
Memory Usage BCCISProf Sony Reader 15 05-16-2007 08:11 PM


All times are GMT -4. The time now is 12:17 PM.


MobileRead.com is a privately owned, operated and funded community.