Possible mem leak while downloading metadata

SoulStyle · 11-13-2011, 07:09 AM

I'm currently trying to repopulate my Tags from Goodreads and thus added like 3600 books to a batch metadata download (only Goodreads as source and only Tags checked).
This attempt completely killed my Windows. Afterwards I tried it in batches of 1000 books. According to the Task-Manager the Calibre Tasks jumps from around 130MB memory footprint to roughly 900MB footprint without releasing the memory after all jobs done.

Kind regards

kovidgoyal · 11-13-2011, 07:11 AM

Update your calibre version.

SoulStyle · 11-13-2011, 07:13 AM

Using 0.8.26 Sorry for not having included this information.

Kind regards

kovidgoyal · 11-13-2011, 07:15 AM

version 0.8.26 batches large metadata downloads into groups of 100 automatically, so I dont see how you could be using 0.8.26

SoulStyle · 11-13-2011, 08:15 AM

True, Calibre separtes the 1000 books in 10 jobs. Nevertheless, each job "costs" like 80MB.
Please see the attached screen-shots:

Calibre_10Jobs: Start of the batch, mem footprint ~260MB
Calibre_02Jobs: 8 jobs have finished processing, mem footprint ~850MB

Kind regards

kovidgoyal · 11-13-2011, 09:41 AM

I cannot replicate that, the memory consumption goes down for me after each job completes (each job keeps the downloaded metadata in memory, when it competes the metadata is written to disk and the memory released).

Since this kind of bulk download is typically a one time thing, I suggest you just do it in a couple of batches. I lack the motivation to try to figure out why the mem isn't being released for you.

SoulStyle · 11-13-2011, 10:19 AM

Do whatever suits you. I was just reporting it

kovidgoyal · 11-13-2011, 11:01 AM

Thanks for the report. I'll keep it in mind the next time I work on that bit of code. For the moment, I just have a lot of stuff that has piled up over the last month that I really want to clear.

SoulStyle · 11-23-2011, 01:24 PM

Update:
I checked a large bulk download with just metadata from Amazon. There was no leak, all memory was nicely released

On a bulk download with just Goodread tags the leak showed like in the posts above.

Kind regards,

SoulStyle

kovidgoyal · 11-23-2011, 09:09 PM

Then you should report it in the goodreads plugin thread.

11-13-2011, 07:09 AM	#1
SoulStyle Enthusiast Posts: 26 Karma: 22 Join Date: May 2011 Device: Kindle 3	Possible mem leak while downloading metadata I'm currently trying to repopulate my Tags from Goodreads and thus added like 3600 books to a batch metadata download (only Goodreads as source and only Tags checked). This attempt completely killed my Windows. Afterwards I tried it in batches of 1000 books. According to the Task-Manager the Calibre Tasks jumps from around 130MB memory footprint to roughly 900MB footprint without releasing the memory after all jobs done. Kind regards

11-13-2011, 08:15 AM	#5
SoulStyle Enthusiast Posts: 26 Karma: 22 Join Date: May 2011 Device: Kindle 3	True, Calibre separtes the 1000 books in 10 jobs. Nevertheless, each job "costs" like 80MB. Please see the attached screen-shots: Calibre_10Jobs: Start of the batch, mem footprint ~260MB Calibre_02Jobs: 8 jobs have finished processing, mem footprint ~850MB Kind regards Attached Thumbnails

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Crash when downloading metadata	meme	Calibre	4	10-14-2011 06:30 AM
Downloading Metadata question -	collin8579	Library Management	6	08-14-2011 09:13 PM
Metadata not downloading	Jay955528	Calibre	4	07-10-2011 02:26 AM
Downloading Metadata question v.08	zed88	Calibre	5	05-18-2011 04:08 AM
0.7.5.7 - Not Downloading Any Metadata	nynaevelan	Calibre	3	04-23-2011 11:38 AM

11-13-2011, 07:11 AM	#2
kovidgoyal creator of calibre Posts: 45,345 Karma: 27182818 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Update your calibre version.

11-13-2011, 07:13 AM	#3
SoulStyle Enthusiast Posts: 26 Karma: 22 Join Date: May 2011 Device: Kindle 3	Using 0.8.26 Sorry for not having included this information. Kind regards

11-13-2011, 07:15 AM	#4
kovidgoyal creator of calibre Posts: 45,345 Karma: 27182818 Join Date: Oct 2006 Location: Mumbai, India Device: Various	version 0.8.26 batches large metadata downloads into groups of 100 automatically, so I dont see how you could be using 0.8.26

11-13-2011, 09:41 AM	#6
kovidgoyal creator of calibre Posts: 45,345 Karma: 27182818 Join Date: Oct 2006 Location: Mumbai, India Device: Various	I cannot replicate that, the memory consumption goes down for me after each job completes (each job keeps the downloaded metadata in memory, when it competes the metadata is written to disk and the memory released). Since this kind of bulk download is typically a one time thing, I suggest you just do it in a couple of batches. I lack the motivation to try to figure out why the mem isn't being released for you.

11-13-2011, 10:19 AM	#7
SoulStyle Enthusiast Posts: 26 Karma: 22 Join Date: May 2011 Device: Kindle 3	Do whatever suits you. I was just reporting it

11-13-2011, 11:01 AM	#8
kovidgoyal creator of calibre Posts: 45,345 Karma: 27182818 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Thanks for the report. I'll keep it in mind the next time I work on that bit of code. For the moment, I just have a lot of stuff that has piled up over the last month that I really want to clear.

11-23-2011, 01:24 PM	#9
SoulStyle Enthusiast Posts: 26 Karma: 22 Join Date: May 2011 Device: Kindle 3	Update: I checked a large bulk download with just metadata from Amazon. There was no leak, all memory was nicely released On a bulk download with just Goodread tags the leak showed like in the posts above. Kind regards, SoulStyle

11-23-2011, 09:09 PM	#10
kovidgoyal creator of calibre Posts: 45,345 Karma: 27182818 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Then you should report it in the goodreads plugin thread.

Advert

Advert