Memory leak?

[cybmole]

Originally Posted by dwanthny

Your right in that once calibre does something it doesn't always release the memory, it seems to cache some data for later use. I'm fine with this.



My metadata.db is 43meg on a library of 9300 books on my finished library and 45megs on my working library. It's not uncommon for calibre to be operating in the 325meg range, but somehow Firefox always seems to use just a little more ram.

so 43+45 + say 100 base, accounts for most of your initial footprint.

and 2.5 + 100 base accounts for most of mine

I have to admit that I only joined the thread out of curiosity - I have no real issue with how much RAM it wants to reserve on this PC , but If I were trying to manage a library like yours on a 1Gb netbook then I could have issues.

[theducks]

XP SP3 2G RAM
Calibre was minimized overnight.
No Tray Icon.
No Fetch News.
Device not attached.

Task manages said 3976K and incrementing.
Maximized: Calibre jumped to high 7xxxK and is still incrementing 4K a step.

CPU% toggles from 00 to 01 occasional between steps.
AVG free is my A/V

Kovid. Is it possible to get a task logger to enable? Not the full debugger.
What the log would show would be user activity:
Minimize, Convert(single), Metaedit(single), Send to.., open Tag Browser, Tag select, Show Book Details,Fetch News (auto) count, Fetch News (manual)
That kind of thing.

Just while I have been writhing this (and nothin has been done with Calibre) it has incremented to 9132K

[kovidgoyal]

Originally Posted by cybmole

i guess the only way forward then is some sort of group user test/poll.

I'm not sure why you are not hearing me. I cannot fix a bug that *I* cannot reproduce.

[kovidgoyal]

Originally Posted by theducks

Kovid. Is it possible to get a task logger to enable? Not the full debugger.
What the log would show would be user activity:
Minimize, Convert(single), Metaedit(single), Send to.., open Tag Browser, Tag select, Show Book Details,Fetch News (auto) count, Fetch News (manual)
That kind of thing.

Just while I have been writhing this (and nothin has been done with Calibre) it has incremented to 9132K

That would be useless, since memory consumption is increasing for you while calibre is not doing anything. That task logger would show absolutely nothing. Debugging this will require some instrumentation to examine the allocated memory. That is too complex to set up for use by anyone other than a developer. Which is why I say again:

I (or indeed another developer) need to be able to reproduce this behavior to have a hope of fixing it.

[cybmole]

Originally Posted by kovidgoyal

I'm not sure why you are not hearing me. I cannot fix a bug that *I* cannot reproduce.

you could maybe reproduce it if the community could give you specific actions that cause RAM footprint to increase, hence the suggestion of a group poll/test as to what usage & behaviour others are seeing.
e.g. O/S, AV, footprint at launch, library size ( may not be relevant), footprint after 12, 24,36... hours, plugins in use, features in use e.g. news....

I assume that you've already tried the obvious i.e. install to a PC, leave that PC switched on for a week, check memory usage each day....

not trying to be rude or anything, but it seems from this thread that everyone *except* you *can* reproduce it to some extent !

in my case though it's only 5MB per hour, or less, while idling; takes a long time to become significant.

[itimpi]

Originally Posted by cybmole

not trying to be rude or anything, but it seems from this thread that everyone *except* you *can* reproduce it to some extent !

Not true! I am running on Windows 7 Pro and cannot reproduce this.

[kovidgoyal]

Originally Posted by cybmole

not trying to be rude or anything, but it seems from this thread that everyone *except* you *can* reproduce it to some extent !

For someone that isn't trying, you're certainly succeeding.

[cybmole]

Originally Posted by itimpi

Not true! I am running on Windows 7 Pro and cannot reproduce this.

see - that's progress. this will only be solved by hearing from both those who can who can & those who can't, & spotting common factors.

e.g. are you using 32 or 64 bit & what AV do you use ?

but I'm going to drop out of this thread now, I have no desire to bug Kovid who does an excellent job. I will however lurk on the thread to see if it ever gets solved.

I will not post any more on this topic unless specifically asked to.

[itimpi]

Originally Posted by cybmole

e.g. are you using 32 or 64 bit & what AV do you use ?

32-bit Windows 7. Microsoft Security Essentials AV.

Win 7 64bit home premium Service pack 1
AMD Athlon
Ram 4GB
AV BT NetProtect Plus from McAfee

Started Calibre last night 20:12 have not used it just minimised.
20:12 7350 K
21:12 25487 K
22:12 47344 K
23:12 50292 K
00:12 54284 K
08:12 86268 K
10:12 92934 K
11:12 98904 K
13:12 105480 K
15:12 188596 K
17:12 120520 K
20:12 139736 K

[jhay777]

What could cause this behaviour on some systems but not others (apart from the obvious OS and AV differences)? Does calibre rely on any runtimes, DLLs, etc that may vary across systems and be the cause of the memory increases that cybmole, onedot3, me and others are seeing?

[chaley]

Originally Posted by jhay777

What could cause this behaviour on some systems but not others (apart from the obvious OS and AV differences)? Does calibre rely on any runtimes, DLLs, etc that may vary across systems and be the cause of the memory increases that cybmole, onedot3, me and others are seeing?

Almost anything. Examples: programs that attempt to communicate between processes, resource monitors, differences in device drivers, hardware differences (memory management & processors), different levels of windows components, different networking configurations, and the like. And don't leave off the possibility of bugs in windows itself triggered by some configurations.

My system (Win7/pro/32) exhibits this 'problem' (actually, I don't care about it so it isn't a problem for me). I just spent close to 10 hours attempting to trace it. Results on my system:

• The problem starts after calibre is open 10 to 20 minutes. Once it starts, the process space grows by 4K every 3 to 5 seconds.
• Calibre itself (python) uses no more memory, even when process size is growing. Calibre can see that the process space has grown, but it has allocated nothing to cause it.
• The problem happens whether or not calibre is minimized to the systray.
• The problem happens whether or not the systray is enabled.
• The problem 'stops' whenever almost anything is touched, such as killing another systray application or clicking on calibre. As far as I can tell, the memory is not reclaimed. The problem restarts in 10 to 20 minutes.
• Killing all interesting systray apps (secunia PSI, dropbox, MS Security essentials, Apache, printer monitors) has no effect other than to sometimes cause the problem to stop for a time.
• - Firefox and Thunderbird both do the same thing on my system, that is from time to time they grow by 4K every few seconds.

Bottom line: this isn't calibre. It might be an interaction between QT and windows, but the fact that the Mozilla apps do it as well argues against that. In any event, there is nothing I can do to fix it, because the problem is almost certainly not inside calibre source code. I give up.

I have shared the experiment details with Kovid in case he has more ideas. I won't hold my breath.

[DoctorOhh]

Originally Posted by chaley

(actually, I don't care about it so it isn't a problem for me).

You and me both.

Originally Posted by chaley

Bottom line: this isn't calibre. It might be an interaction between QT and windows, but the fact that the Mozilla apps do it as well argues against that. In any event, there is nothing I can do to fix it, because the problem is almost certainly not inside calibre source code. I give up.

Thanks for the analysis and your time. I'm always a little amused that no matter how large calibre gets when using it Mozilla always seems to be a little larger.

[jhay777]

Excellent summary of the issue chaley. For what its worth I don't have the same problem with Firefox.

Point taken that this may be the problem I'm having when I'm not having a problem, but it would be good to know what is causing this behaviour and why. We still have no idea what is causing calibre to gobble up 600MB+ doing nothing!

Whatever happens, I believe it would be worth documenting that (on systems that see unusually large and increasing memory use) calibre needs to be touched every day or so if left running minimised in the task bar or in the system tray. Because that is what I'm going to have to do on my systems that are running calibre.

Something I have noticed

Win Xp 32bit using the same library as on my win7 64bit
After starting, calibre was using 135734 K full screen
Minimise and calibre was using 4638 K
Maximise and calibre was using 10876 K

On win7 64 bit
minimising and maximising there is no change
calibre is now using 231036 K

It may be something on Win7 64bit as I use O&O clever cache 64Bit
Which is for controlling memory
It says that it has recoverd 112MB but the program is now using 394MB
When it is started it is using around 22MB but this keeps growing and growing.
O&O clever cache has not been running on win7 whilst I was getting readings
But was running on Win XP so it is not O&O clever cache
Both machines run BT Netprotect Plus by McAfee does not look like it is the AV

Win7 64bit
I have just switched to a small library of approx 500 books calibre has now gone up to 266376 K