Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 03-05-2016, 09:32 PM   #1
Dapper
Member
Dapper began at the beginning.
 
Dapper's Avatar
 
Posts: 11
Karma: 10
Join Date: Apr 2010
Device: None
Calibre memory usage - Linux

I'm currently in the process of editing metadata in my newly created Calibre Library (approximately 1500 books) and I noticed the memory utilisation seemed to be increasing gradually.

Upon loading Calibre the initial memory used is:
Code:
22504 python2 /usr/bin/calibre-parallel --pipe-worker from calibre.utils.ipc.pool import run_main, worker_main; run_main(worker_main)
681200 python2 /usr/bin/calibre --detach
After 20 minutes or so:
Code:
22504 python2 /usr/bin/calibre-parallel --pipe-worker from calibre.utils.ipc.pool import run_main, worker_main; run_main(worker_main)
70620 python2 /usr/bin/calibre-parallel
71468 python2 /usr/bin/calibre-parallel
827796 python2 /usr/bin/calibre --detach
After approximately 40 minutes:
Code:
22504 python2 /usr/bin/calibre-parallel --pipe-worker from calibre.utils.ipc.pool import run_main, worker_main; run_main(worker_main)
1056120 python2 /usr/bin/calibre --detach
The memory keeps increasing slowly beyond this.

So far I have used the copy of Calibre in the repos (Arch based Manjaro) and the binary install from the Calibre web site. I was thinking about building from git but I thought I'd check here first.

Is this a memory leak?
Dapper is offline   Reply With Quote
Old 03-05-2016, 10:33 PM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,776
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
No it isn't. Dynamic languages like python keep memory around in memory pools rather than always returning it to the OS immediately, for performance reasons.
kovidgoyal is offline   Reply With Quote
Advert
Old 03-05-2016, 10:39 PM   #3
Dapper
Member
Dapper began at the beginning.
 
Dapper's Avatar
 
Posts: 11
Karma: 10
Join Date: Apr 2010
Device: None
Quote:
Originally Posted by kovidgoyal View Post
No it isn't. Dynamic languages like python keep memory around in memory pools rather than always returning it to the OS immediately, for performance reasons.
So, the memory increase from ~700MB at start to 1.5GB currently is normal?
Dapper is offline   Reply With Quote
Old 03-05-2016, 10:43 PM   #4
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,776
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
According to the figures you posted the memory increase is from

700MB

to

1GB

that is normal.

If you are saying that memory usage keeps increasing unboundedly then that is not normal and is almost certainly something specific to your system. Start by disabling any third party plugins you have installed.
kovidgoyal is offline   Reply With Quote
Old 03-05-2016, 10:50 PM   #5
Dapper
Member
Dapper began at the beginning.
 
Dapper's Avatar
 
Posts: 11
Karma: 10
Join Date: Apr 2010
Device: None
Quote:
Originally Posted by kovidgoyal View Post
According to the figures you posted the memory increase is from

700MB

to

1GB

that is normal.

If you are saying that memory usage keeps increasing unboundedly then that is not normal and is almost certainly something specific to your system. Start by disabling any third party plugins you have installed.
The memory has increased from ~1GB in my original post to ~1.5GB now. This is using Calibre out of the box, no plug-ins. I'm just about to install in a VM using the same base OS (Xfce)
Dapper is offline   Reply With Quote
Advert
Old 03-05-2016, 11:02 PM   #6
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,776
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
What would be the point of running in it in a vm with the same system? Use a different system.

Also what are you doing in calibre to cause the memory to increase? For instance if I start calibre memory usage is ~ 100MB then I use the edit metadata dialog for many books, memory usage climbs to 200MB and stays constant there no matter how many books I edit the metadata for. Since most tasks in calibre happen in short lived worker processes, I dont see how they could be leaking memory.

The only other things I can think of that happen in the same process are running the calibre content server and connecting to a device.

Two things you can do to decrease coupling to the system is use the binary calibre install and in Preferences->Look & Feel change the theme from system default to calibre.
kovidgoyal is offline   Reply With Quote
Old 03-05-2016, 11:24 PM   #7
Dapper
Member
Dapper began at the beginning.
 
Dapper's Avatar
 
Posts: 11
Karma: 10
Join Date: Apr 2010
Device: None
Quote:
Originally Posted by kovidgoyal View Post
What would be the point of running in it in a vm with the same system? Use a different system.
To see if the issue is caused by something I've install in Xfce that isn't present in a clean install of the same base system. No point using Windows or Debian, it;s sticks and stones.

Quote:
Also what are you doing in calibre to cause the memory to increase? For instance if I start calibre memory usage is ~ 100MB then I use the edit metadata dialog for many books, memory usage climbs to 200MB and stays constant there no matter how many books I edit the metadata for. Since most tasks in calibre happen in short lived worker processes, I dont see how they could be leaking memory.
As mentioned in my original post, I'm just editing metadata, in some cases individual books and occasionally several books at a time. I've not selected more than 10 books simultaneously.

Quote:
The only other things I can think of that happen in the same process are running the calibre content server and connecting to a device.
The server isn't currently running, I don't have wireless on my desktop and there ar no devices selected or connected.

Quote:
Two things you can do to decrease coupling to the system is use the binary calibre install and in Preferences->Look & Feel change the theme from system default to calibre.
I mentioned in my original post that I see the same issues with the version of Calibre from the distro repo and the binary install from the Calibre website.

For what it's worth, with a clean install of the OS and a clean install of Calibre - no library defined - the memory is:

Code:
16592 python2 /usr/bin/calibre-parallel --pipe-worker from calibre.utils.ipc.pool import run_main, worker_main; run_main(worker_main)
254932 python2 /usr/bin/calibre --detach
Dapper is offline   Reply With Quote
Old 03-05-2016, 11:50 PM   #8
Dapper
Member
Dapper began at the beginning.
 
Dapper's Avatar
 
Posts: 11
Karma: 10
Join Date: Apr 2010
Device: None
It appears to be something on my main system causing this memory climb as importing my library into the new VM based system only causes a minimal change (lower?) in memory:

Code:
16468 python2 /usr/bin/calibre-parallel --pipe-worker from calibre.utils.ipc.pool import run_main, worker_main; run_main(worker_main)
249796 python2 /usr/bin/calibre
Question, are python libraries shared between running applications?

Last edited by Dapper; 03-05-2016 at 11:53 PM.
Dapper is offline   Reply With Quote
Old 03-05-2016, 11:53 PM   #9
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,776
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Pure python libraries are not. Python libraries compiled to native code are shared, just like other native libraries.
kovidgoyal is offline   Reply With Quote
Old 03-05-2016, 11:59 PM   #10
Dapper
Member
Dapper began at the beginning.
 
Dapper's Avatar
 
Posts: 11
Karma: 10
Join Date: Apr 2010
Device: None
Ok, thanks. I'll investigate and maybe reinstall.
Dapper is offline   Reply With Quote
Old 03-07-2016, 10:51 PM   #11
Dapper
Member
Dapper began at the beginning.
 
Dapper's Avatar
 
Posts: 11
Karma: 10
Join Date: Apr 2010
Device: None
I found a cause for the massive increase in initial memory, changing the icons. In all honesty, in my earlier post, I'd forgotten I'd changed the icons, it wasn't until I started playing around in a VM that I noticed.

From what I can see, Circle UI and Square UI are the main culprits, both of which increase initial memory by several hundred megabytes.

Initial memory - default icons
Code:
22480  9684 python2 /usr/bin/calibre-parallel --pipe-worker from calibre.utils.ipc.pool import run_main, worker_main; run_main(worker_main)
246860 9656 python2 /usr/bin/calibre
Memory after changing to Square UI and restarting:
Code:
22292  9434 python2 /usr/bin/calibre-parallel --pipe-worker from calibre.utils.ipc.pool import run_main, worker_main; run_main(worker_main)
672080 9406 python2 /usr/bin/calibre
Reset to default icons:
Code:
22288  9790 python2 /usr/bin/calibre-parallel --pipe-worker from calibre.utils.ipc.pool import run_main, worker_main; run_main(worker_main)
251532 9758 python2 /usr/bin/calibre
Change to Circle UI and restart:
Code:
22292  9887 python2 /usr/bin/calibre-parallel --pipe-worker from calibre.utils.ipc.pool import run_main, worker_main; run_main(worker_main)
708424 9851 python2 /usr/bin/calibre
I also tried the Material icon set but there was no significant change in memory.
Dapper is offline   Reply With Quote
Old 03-07-2016, 11:03 PM   #12
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,776
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Those two icon themes presumably use very high resolution images as icons. You might want to post in the threads for the icon themes asking their creator to reduce the resolution of the images.
kovidgoyal is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
ebook-convert: reduce memory usage Loteckal Related Tools 2 04-16-2015 12:52 AM
n2a card memory usage jvik Android Devices 1 04-10-2015 02:28 PM
Memory usage jessie102 Calibre 4 01-29-2015 12:44 PM
Extreme memory usage MJBoa Calibre 1 02-12-2010 11:37 AM
Memory Usage BCCISProf Sony Reader 15 05-16-2007 08:11 PM


All times are GMT -4. The time now is 08:41 PM.


MobileRead.com is a privately owned, operated and funded community.