Memory leak in 0.8.28?

kongzifan · 12-01-2011, 11:44 AM

After upgrading to 0.8.28, calibre became significantly slow. When I tried to convert downloaded 6M python 3.2.2 HTML document to mobi, it took more than one hour on an dual core desktop. Calibre used up 1.9G of the 2G memory and 1.4G of the 4G swap, and the computer became extremely slow and unresponsive. Is there a memory leak problem in 0.8.28?

By the way, the system is Ubuntu 10.04 64bit, CPU is dual core AMD at 2.7G Hz,and total memory is 2G.

theducks · 12-01-2011, 11:55 AM

Quote:

Originally Posted by kongzifan

After upgrading to 0.8.28, calibre became significantly slow. When I tried to convert downloaded 6M python 3.2.2 HTML document to mobi, it took more than one hour on an dual core desktop. Calibre used up all the 2G memory and the computer became extremely slow and unresponsive. Is there a memory leak problem in 0.8.28?

Heuristics ground a simple TXT=>EPUB file to a standstill (2 hrs before I aborted). 50% (the max for a HT P4) CPU, no I/O.

1 minute with Heuristics OFF.

kongzifan · 12-01-2011, 12:05 PM

Heuristics is already off.

kovidgoyal · 12-01-2011, 12:36 PM

In order to conclude that 0.8.28 had a memory leak you would need to demonstrate that a conversion that did not use memory in 0.8.27 now uses memory in 0.8.28. Some conversions take a lot of memory. That is the nature of the beast. Convert a file you converted in 0.8.27 in 0.8.28, I highly doubt you will find memory usage any higher.

kongzifan · 12-02-2011, 04:13 AM

I got this impression because this calibre making system unresponsive thing never happened before. I had used calibre to convert many much larger documents without having this trouble. But I did not make deliberate comparison of different versions of calibre on the same document, thus it's not conclusive.

kovidgoyal · 12-02-2011, 05:19 AM

Memory usage has nothing to do with size. I can artificially construct a 10MB document that would consume 4GB of RAM while converting, while a more "normal" 10MB document typically takes no more than a 100MB.

In the case of the python documentation, the document has the structure of a lot of relatively small HTML files with a single very large CSS that is referenced by all of them. In such a case, because of calibre's CSS flattening/font size normalization, the CSS has to be duplicated for every HTML file. Running a calibre conversion on a collection of HTML files is like opening every single one simultaneously in a different tab in a single browser.

kongzifan · 12-02-2011, 09:46 AM

I see. That makes a lot of sense. Thank you.

12-01-2011, 11:44 AM	#1
kongzifan Enthusiast Posts: 27 Karma: 10 Join Date: Mar 2009 Device: Kindle, iPhone, iPad	Memory leak in 0.8.28? After upgrading to 0.8.28, calibre became significantly slow. When I tried to convert downloaded 6M python 3.2.2 HTML document to mobi, it took more than one hour on an dual core desktop. Calibre used up 1.9G of the 2G memory and 1.4G of the 4G swap, and the computer became extremely slow and unresponsive. Is there a memory leak problem in 0.8.28? By the way, the system is Ubuntu 10.04 64bit, CPU is dual core AMD at 2.7G Hz,and total memory is 2G. Last edited by kongzifan; 12-01-2011 at 11:58 AM. Reason: added system specs

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Possible mem leak while downloading metadata	SoulStyle	Calibre	9	11-23-2011 09:09 PM
Memory leak?	jhay777	Calibre	45	10-06-2011 10:18 AM
Kindle 3 Screen Leak??	LuBiB	Amazon Kindle	59	06-04-2011 03:48 PM
[Old Thread] Memory leak, Worker procees closeing	theducks	Calibre	14	03-30-2011 09:06 PM

12-01-2011, 12:05 PM	#3
kongzifan Enthusiast Posts: 27 Karma: 10 Join Date: Mar 2009 Device: Kindle, iPhone, iPad	Heuristics is already off.

12-01-2011, 12:36 PM	#4
kovidgoyal creator of calibre Posts: 43,850 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	In order to conclude that 0.8.28 had a memory leak you would need to demonstrate that a conversion that did not use memory in 0.8.27 now uses memory in 0.8.28. Some conversions take a lot of memory. That is the nature of the beast. Convert a file you converted in 0.8.27 in 0.8.28, I highly doubt you will find memory usage any higher.

12-02-2011, 04:13 AM	#5
kongzifan Enthusiast Posts: 27 Karma: 10 Join Date: Mar 2009 Device: Kindle, iPhone, iPad	I got this impression because this calibre making system unresponsive thing never happened before. I had used calibre to convert many much larger documents without having this trouble. But I did not make deliberate comparison of different versions of calibre on the same document, thus it's not conclusive.

12-02-2011, 05:19 AM	#6
kovidgoyal creator of calibre Posts: 43,850 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Memory usage has nothing to do with size. I can artificially construct a 10MB document that would consume 4GB of RAM while converting, while a more "normal" 10MB document typically takes no more than a 100MB. In the case of the python documentation, the document has the structure of a lot of relatively small HTML files with a single very large CSS that is referenced by all of them. In such a case, because of calibre's CSS flattening/font size normalization, the CSS has to be duplicated for every HTML file. Running a calibre conversion on a collection of HTML files is like opening every single one simultaneously in a different tab in a single browser.

12-02-2011, 09:46 AM	#7
kongzifan Enthusiast Posts: 27 Karma: 10 Join Date: Mar 2009 Device: Kindle, iPhone, iPad	I see. That makes a lot of sense. Thank you.

Advert

Advert