Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 01-21-2011, 01:28 AM   #1
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,781
Karma: 12516053
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Run Away Memory or Expected Results?

While working in calibre, moving between two libraries there is an ever increasing memory footprint every time I switch between the libraries.

When I switch to my 5000 book epub library memory use increases 25M. When I switch to my 12500 book mixed format library memory use increases by 120M.

At no point does calibre release any of the memory. Currently memory use is sitting at 1G and the only way to reduce this number is to restart calibre.

I'm just curious if this is a known expected outcome of running multiple libraries.
DoctorOhh is offline   Reply With Quote
Old 01-21-2011, 01:36 AM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,446
Karma: 4961459
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
No, it's a memory leak. It's a rather hard to track down leak however, as references to the library are held in a *lot* of places, which is why I haven't bothered fixing it as yet.
kovidgoyal is offline   Reply With Quote
Old 01-21-2011, 01:48 AM   #3
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,781
Karma: 12516053
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Quote:
Originally Posted by kovidgoyal View Post
No, it's a memory leak. It's a rather hard to track down leak however, as references to the library are held in a *lot* of places, which is why I haven't bothered fixing it as yet.
Thanks for the quick response. My normal work flow is to use one library at a time so this memory leak hasn't been a problem.

The other day calibre died (a rare occurrence) on my work machine and I noted it was over 500M in size at the time. I had been testing two GUI plugins that Kiwidude is about to release so I started poking around to try and figure out where the increased memory came from. I was glad to see it wasn't from the plugins.

Thanks again for all your work, don't waste your time on this leak it isn't a major problem, I was just curious.
DoctorOhh is offline   Reply With Quote
Old 01-21-2011, 11:49 AM   #4
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,446
Karma: 4961459
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
I found and fixed one leak. There may still be problems depending on what you are doing in calibre, for example running the content server. Now switching back and forth between libraries will not cause an increase in mem usage.
kovidgoyal is offline   Reply With Quote
Old 01-21-2011, 12:53 PM   #5
paulfiera
Addict
paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.
 
paulfiera's Avatar
 
Posts: 339
Karma: 3102
Join Date: Dec 2010
Location: EU
Device: iPad, Kobo Glo
Quote:
Originally Posted by dwanthny View Post
When I switch to my 12500 book mixed format library...
Walt, how's performance like with a library with 12000+ books?

I ask this because, for performance reasons - and maybe I'm wrong - I've splitted my main library in two. One with about 1000 books and the other with about 800.

I'd rather have everything in one library but I've been worried about performance.
paulfiera is offline   Reply With Quote
Old 01-21-2011, 02:04 PM   #6
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by paulfiera View Post
Walt, how's performance like with a library with 12000+ books?
I have no performance problems using a 14,000 book library.
Starson17 is offline   Reply With Quote
Old 01-21-2011, 04:44 PM   #7
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,781
Karma: 12516053
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Quote:
Originally Posted by paulfiera View Post
Walt, how's performance like with a library with 12000+ books?
6 months ago there were all sort of performance problems when updating bulk metadata, tags etc... but these issues have all been resolved and there is no noticeable performance hit.
DoctorOhh is offline   Reply With Quote
Old 01-21-2011, 05:31 PM   #8
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,781
Karma: 12516053
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
FYI, I'm using Windows XP sp3 with calibre 0.7.41

Quote:
Originally Posted by kovidgoyal View Post
I found and fixed one leak.

Now switching back and forth between libraries will not cause an increase in mem usage.
Doing nothing more than switching back an forth between libraries still increases memory usage, by slightly less per switch than previously stated, that memory is never released by calibre.

Like you said,

Quote:
It's a rather hard to track down leak however, as references to the library are held in a *lot* of places
I see this causing rare problems unless someone specifically tries to make it cause a problem.

One leak down xxx leaks left to go.
DoctorOhh is offline   Reply With Quote
Old 01-21-2011, 06:08 PM   #9
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,446
Karma: 4961459
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Hmm can't replicate that. To confirm, all you did was start calibre and repeatedly switch libraries?
kovidgoyal is offline   Reply With Quote
Old 01-21-2011, 06:24 PM   #10
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,781
Karma: 12516053
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Quote:
Originally Posted by kovidgoyal View Post
Hmm can't replicate that. To confirm, all you did was start calibre and repeatedly switch libraries?
Yes, that is exactly what I did. I use this bat file to run calibre.

Spoiler:
Code:
cd c:\
rmdir /s /q c:\Calibre_temp

REM Create new temp directory for this session.

cd c:\
mkdir Calibre_temp

REM Specify the location of the Calibre temp files
REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

SET CALIBRE_TEMP_DIR=C:\Calibre_temp

REM        CalibreRun.bat
REM        ~~~~~~~~~~~~~~
REM
REM Batch File to start a Calibre configuration on Windows
REM giving explicit control of the location of:

REM  - Calibre Config Files

REM Specify the location of the Calibre Config files
REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

SET CALIBRE_CONFIG_DIRECTORY=C:\My Dropbox\CalibreConfig

REM Location of calibre program files
REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

C:
CD \Program Files\Calibre2

REM Display settings that will be used
REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

echo CONFIG=%CALIBRE_CONFIG_DIRECTORY%

REM  The following gives a chance to check the settings before
REM  starting Calibre.  It can be commented out if not wanted.

echo "Press CTRL-C if you do not want to continue"
pause

REM Start up the calibre program.
REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
REM

start calibre.exe --start-in-tray

Update:
  • Memory initial load of 12500 book library - 173M
  • Memory after 1st switch to 5000 book library - 207M
  • Memory after 2nd switch back to 12500 book library - 319M
  • Memory after 3rd switch back to 5000 book library - 354M
  • Memory after 4th switch back to 12500 book library - 456M

Last edited by DoctorOhh; 01-21-2011 at 06:45 PM.
DoctorOhh is offline   Reply With Quote
Old 01-21-2011, 08:11 PM   #11
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,446
Karma: 4961459
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
And do you have the calibre content server running?
kovidgoyal is offline   Reply With Quote
Old 01-21-2011, 10:21 PM   #12
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,781
Karma: 12516053
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Quote:
Originally Posted by kovidgoyal View Post
And do you have the calibre content server running?
No, I just open calibre using the bat file above and switch from one library to the other to demo the leak.
DoctorOhh is offline   Reply With Quote
Old 01-22-2011, 01:33 AM   #13
paulfiera
Addict
paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.
 
paulfiera's Avatar
 
Posts: 339
Karma: 3102
Join Date: Dec 2010
Location: EU
Device: iPad, Kobo Glo
Thanks, dwanthny and Starson17.

I think I'm gonna merge 2 of my libraries.
paulfiera is offline   Reply With Quote
Old 01-22-2011, 08:46 AM   #14
GRiker
Comparer of the Ephemeris
GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.GRiker ought to be getting tired of karma fortunes by now.
 
Posts: 1,497
Karma: 424627
Join Date: Mar 2009
Device: iPad
I'm seeing a memory leak as well, and I'm not running the content server. 0.7.42, OSX.

Initial load, library 1: 247MB
Switch to library 2: 270MB
Switch to library 1: 277.5MB
Switch to library 2: 296.5MB
Switch to library 1: 303.7MB

...

G
GRiker is offline   Reply With Quote
Old 01-22-2011, 01:05 PM   #15
paulfiera
Addict
paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.paulfiera could sell banana peel slippers to a Deveel.
 
paulfiera's Avatar
 
Posts: 339
Karma: 3102
Join Date: Dec 2010
Location: EU
Device: iPad, Kobo Glo
Quote:
Originally Posted by dwanthny View Post
...I use this bat file to run calibre.

Spoiler:
Code:
cd c:\
rmdir /s /q c:\Calibre_temp

REM Create new temp directory for this session.

cd c:\
mkdir Calibre_temp

REM Specify the location of the Calibre temp files
REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

SET CALIBRE_TEMP_DIR=C:\Calibre_temp

REM        CalibreRun.bat
REM        ~~~~~~~~~~~~~~
REM
REM Batch File to start a Calibre configuration on Windows
REM giving explicit control of the location of:

REM  - Calibre Config Files

REM Specify the location of the Calibre Config files
REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

SET CALIBRE_CONFIG_DIRECTORY=C:\My Dropbox\CalibreConfig

REM Location of calibre program files
REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

C:
CD \Program Files\Calibre2

REM Display settings that will be used
REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

echo CONFIG=%CALIBRE_CONFIG_DIRECTORY%

REM  The following gives a chance to check the settings before
REM  starting Calibre.  It can be commented out if not wanted.

echo "Press CTRL-C if you do not want to continue"
pause

REM Start up the calibre program.
REM ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
REM

start calibre.exe --start-in-tray
dwanthny, even when you set the environment variable CALIBRE_TEMP_DIR, calibre will still create files and folders in your system's tmp enviroment variable.

Folders fontconfig and gen_py and several calxx.tmp files. Also, if you have "open in ePub Editor" in your context menu, a folder "Sigil" - if Sigil is your ePub editor - will be created in this folder.

The only way right now to make caliber put all its temp files in a given directory is to use both the TEMP and TMP system variables.

My calibre batch file starts like this:

Code:
if not exist "d:\calibre" md "d:\calibre" || goto error
SET TEMP=d:\calibre
SET TMP=d:\calibre
Kovid: maybe we could improve CALIBRE_TEMP_DIR to place all the temp files in the directory defined by this env variable?

Last edited by paulfiera; 01-22-2011 at 01:16 PM.
paulfiera is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
No. of visitors expected Manishsb General Discussions 5 08-05-2010 11:20 AM
The Kindle Previewer expected soon Sheikspeare Amazon Kindle 2 07-04-2010 12:34 PM
Run Sony 505 software from memory stick? Technophobe Sony Reader 3 12-07-2008 03:51 PM


All times are GMT -4. The time now is 02:06 PM.


MobileRead.com is a privately owned, operated and funded community.