Run Away Memory or Expected Results?

[DoctorOhh]

While working in calibre, moving between two libraries there is an ever increasing memory footprint every time I switch between the libraries.

When I switch to my 5000 book epub library memory use increases 25M. When I switch to my 12500 book mixed format library memory use increases by 120M.

At no point does calibre release any of the memory. Currently memory use is sitting at 1G and the only way to reduce this number is to restart calibre.

I'm just curious if this is a known expected outcome of running multiple libraries.

[kovidgoyal]

No, it's a memory leak. It's a rather hard to track down leak however, as references to the library are held in a *lot* of places, which is why I haven't bothered fixing it as yet.

[DoctorOhh]

Originally Posted by kovidgoyal

No, it's a memory leak. It's a rather hard to track down leak however, as references to the library are held in a *lot* of places, which is why I haven't bothered fixing it as yet.

Thanks for the quick response. My normal work flow is to use one library at a time so this memory leak hasn't been a problem.

The other day calibre died (a rare occurrence) on my work machine and I noted it was over 500M in size at the time. I had been testing two GUI plugins that Kiwidude is about to release so I started poking around to try and figure out where the increased memory came from. I was glad to see it wasn't from the plugins.

Thanks again for all your work, don't waste your time on this leak it isn't a major problem, I was just curious.

[kovidgoyal]

I found and fixed one leak. There may still be problems depending on what you are doing in calibre, for example running the content server. Now switching back and forth between libraries will not cause an increase in mem usage.

[paulfiera]

Originally Posted by dwanthny

When I switch to my 12500 book mixed format library...

Walt, how's performance like with a library with 12000+ books?

I ask this because, for performance reasons - and maybe I'm wrong - I've splitted my main library in two. One with about 1000 books and the other with about 800.

I'd rather have everything in one library but I've been worried about performance.

[Starson17]

Originally Posted by paulfiera

Walt, how's performance like with a library with 12000+ books?

I have no performance problems using a 14,000 book library.

[DoctorOhh]

Originally Posted by paulfiera

Walt, how's performance like with a library with 12000+ books?

6 months ago there were all sort of performance problems when updating bulk metadata, tags etc... but these issues have all been resolved and there is no noticeable performance hit.

[DoctorOhh]

FYI, I'm using Windows XP sp3 with calibre 0.7.41

Originally Posted by kovidgoyal

I found and fixed one leak.

Now switching back and forth between libraries will not cause an increase in mem usage.

Doing nothing more than switching back an forth between libraries still increases memory usage, by slightly less per switch than previously stated, that memory is never released by calibre.

Like you said,

It's a rather hard to track down leak however, as references to the library are held in a *lot* of places

I see this causing rare problems unless someone specifically tries to make it cause a problem.

One leak down xxx leaks left to go.

[kovidgoyal]

Hmm can't replicate that. To confirm, all you did was start calibre and repeatedly switch libraries?

[DoctorOhh]

Originally Posted by kovidgoyal

Hmm can't replicate that. To confirm, all you did was start calibre and repeatedly switch libraries?

Yes, that is exactly what I did. I use this bat file to run calibre.

Spoiler:

cd c:\
rmdir /s /q c:\Calibre_temp

REM Create new temp directory for this session.

cd c:\
mkdir Calibre_temp

REM Specify the location of the Calibre temp files
REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

SET CALIBRE_TEMP_DIR=C:\Calibre_temp

REM        CalibreRun.bat
REM        ~~~~~~~~~~~~~~
REM
REM Batch File to start a Calibre configuration on Windows
REM giving explicit control of the location of:

REM  - Calibre Config Files

REM Specify the location of the Calibre Config files
REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

SET CALIBRE_CONFIG_DIRECTORY=C:\My Dropbox\CalibreConfig

REM Location of calibre program files
REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

C:
CD \Program Files\Calibre2

REM Display settings that will be used
REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

echo CONFIG=%CALIBRE_CONFIG_DIRECTORY%

REM  The following gives a chance to check the settings before
REM  starting Calibre.  It can be commented out if not wanted.

echo "Press CTRL-C if you do not want to continue"
pause

REM Start up the calibre program.
REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
REM

start calibre.exe --start-in-tray

Update:

• Memory initial load of 12500 book library - 173M
• Memory after 1st switch to 5000 book library - 207M
• Memory after 2nd switch back to 12500 book library - 319M
• Memory after 3rd switch back to 5000 book library - 354M
• Memory after 4th switch back to 12500 book library - 456M

[kovidgoyal]

And do you have the calibre content server running?

[DoctorOhh]

Originally Posted by kovidgoyal

And do you have the calibre content server running?

No, I just open calibre using the bat file above and switch from one library to the other to demo the leak.

[paulfiera]

Thanks, dwanthny and Starson17.

I think I'm gonna merge 2 of my libraries.

[GRiker]

I'm seeing a memory leak as well, and I'm not running the content server. 0.7.42, OSX.

Initial load, library 1: 247MB
Switch to library 2: 270MB
Switch to library 1: 277.5MB
Switch to library 2: 296.5MB
Switch to library 1: 303.7MB

...

G

[paulfiera]

Originally Posted by dwanthny

...I use this bat file to run calibre.

Spoiler:

cd c:\
rmdir /s /q c:\Calibre_temp

REM Create new temp directory for this session.

cd c:\
mkdir Calibre_temp

REM Specify the location of the Calibre temp files
REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

SET CALIBRE_TEMP_DIR=C:\Calibre_temp

REM        CalibreRun.bat
REM        ~~~~~~~~~~~~~~
REM
REM Batch File to start a Calibre configuration on Windows
REM giving explicit control of the location of:

REM  - Calibre Config Files

REM Specify the location of the Calibre Config files
REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

SET CALIBRE_CONFIG_DIRECTORY=C:\My Dropbox\CalibreConfig

REM Location of calibre program files
REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

C:
CD \Program Files\Calibre2

REM Display settings that will be used
REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

echo CONFIG=%CALIBRE_CONFIG_DIRECTORY%

REM  The following gives a chance to check the settings before
REM  starting Calibre.  It can be commented out if not wanted.

echo "Press CTRL-C if you do not want to continue"
pause

REM Start up the calibre program.
REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
REM

start calibre.exe --start-in-tray

dwanthny, even when you set the environment variable CALIBRE_TEMP_DIR, calibre will still create files and folders in your system's tmp enviroment variable.

Folders fontconfig and gen_py and several calxx.tmp files. Also, if you have "open in ePub Editor" in your context menu, a folder "Sigil" - if Sigil is your ePub editor - will be created in this folder.

The only way right now to make caliber put all its temp files in a given directory is to use both the TEMP and TMP system variables.

My calibre batch file starts like this:

if not exist "d:\calibre" md "d:\calibre" || goto error
SET TEMP=d:\calibre
SET TMP=d:\calibre

Kovid: maybe we could improve CALIBRE_TEMP_DIR to place all the temp files in the directory defined by this env variable?