Calibre v7.33 Dramatically Slower - Page 2

paulfiera · 12-15-2010, 01:46 PM

Quote:

Originally Posted by Worldwalker

And this is why calibre rocks. How many other programs have the author saying "It'll be fixed Friday"? The words "working as intended" and "next year's version" come to mind.

+1

kovidgoyal · 12-15-2010, 02:14 PM

@ldolse: The fix for the mem leak in the edit metadata dialog has already been committed. Are you still seeing the memory usage of calibre climb as time passes (after adding a book)?

paulfiera · 12-15-2010, 02:28 PM

Quote:

Originally Posted by kovidgoyal

The fix for the mem leak in the edit metadata dialog has already been committed. Are you still seeing the memory usage of calibre climb as time passes (after adding a book)?

Not here.

There were some crashes in 0.732 after adding several covers from the edit metadata dialog, or after cancelling the dialogbox in the "Fecht metadata from server" if the update took long time. But that seems to have been solved in 0.7.33.

ldolse · 12-15-2010, 02:42 PM

Quote:

Originally Posted by kovidgoyal

@ldolse: The fix for the mem leak in the edit metadata dialog has already been committed. Are you still seeing the memory usage of calibre climb as time passes (after adding a book)?

Looks like it - memory usage was at 218 megs when I checked it after adding one book..
Came back a bit later and it was 222.
Added three books and it jumped up to 231.
Clicked on a bunch of individual books one by one in the main window, and it jumped up to 246 (this dropped back to 232 after a couple minutes though, I think this last bit is fine).

I'm synced from trunk as of about an hour ago.

kovidgoyal · 12-15-2010, 03:59 PM

Quote:

Originally Posted by ldolse

Looks like it - memory usage was at 218 megs when I checked it after adding one book..
Came back a bit later and it was 222.
Added three books and it jumped up to 231.
Clicked on a bunch of individual books one by one in the main window, and it jumped up to 246 (this dropped back to 232 after a couple minutes though, I think this last bit is fine).

I'm synced from trunk as of about an hour ago.

I committed some more mem leak fixes, try again. Also what format are the files you are adding?

ldolse · 12-15-2010, 05:00 PM

Will try in a few hours, laptop power supply died, need to get a replacement. I was adding mostly epub, but there was also a mobi file. Metadata was being pulled from the book's internal metadata when adding rather than the filename.

kovidgoyal · 12-15-2010, 07:57 PM

Quote:

Originally Posted by ldolse

Will try in a few hours, laptop power supply died, need to get a replacement. I was adding mostly epub, but there was also a mobi file. Metadata was being pulled from the book's internal metadata when adding rather than the filename.

Note that *some* increase in memory usage when adding books is expected (the metadata for the books is stored in memory, just like for the rest of the books) but it should be less than an MB per book certainly. Also when you select different books in the list the memory usage can go up and down (this is because calibre maintains an in memory cover cache of the 50 books closest to the current book in the book list).

ldolse · 12-15-2010, 10:06 PM

Yes, I saw that upon further testing. It turned out that I was loading more than just epub and mobi before, there were also some lit and lrf books in the group that I'd randomly grabbed for testing. Will initializing the different input plugins and metadata readers for each format use up some additional memory?

Calibre started up using 209 megs of memory, and after adding a number of books (12 different titles, various formats) it rose up to 247-249 megs of memory (depending on which test run). It seemed like the memory usage stabilized somewhat at that point, but then I was adding and deleting the same group of books.

Different formats seem to use varying amounts of memory, I can do some more testing around that.

Note deleting all the books which caused the increase in memory didn't seem to free up any of that memory. Probably not a big deal, as that's not a wide use case.

I'll also let it run overnight again - before I was seeing the slowness after letting it sit open for several hours.

kovidgoyal · 12-15-2010, 10:18 PM

If you still get the slowdown tomorrow, then I'd be concerned. If you do, report the memory usage.

ldolse · 12-16-2010, 03:14 PM

Seems like it's better today, though it was quite slow when I initially accessed it. I think one of the things that was contributing to the perception of slowness is that Calibre gets paged out of 'real' memory to Virtual Memory. It takes a long time to get everything back into real memory, and during that period the UI is extremely sluggish for a good 10-15 seconds. After that adding books is quite slow for the first instance of each input format - the first LRF book I added took 3.6 seconds. Each additional format was similarly slow, but once all the different input plugin code had been 'exercised' everything was back to the speeds I was seeing the yesterday.

I don't recall previous releases being so slow after leaving the program alone for a while. Did something change with the way things are paged in and out of virtual memory?

Memory use was 154 megs, and jumped to 209 after using it a bit, so I don't think any leaks are the culprit.

kovidgoyal · 12-16-2010, 03:23 PM

I can't think of anything that could cause more paging in newer releases. That said, in fixing memory leaks I have changed the way object references are stored/garbage collected in various parts of the calibre code and that might be causing this, though I don't know enough about the internals of python's garbage collection/windows memory management to speak with any confidence.

As long as the speed recovers after the initial add of a format, I'm not going to spend more time on this. I will note that I don't notice any post idling slowdown on my linux machine but then that has 6 GB of RAM and linux never pages anything unless it really has to.

ldolse · 12-16-2010, 03:27 PM

Yeah, it definitely seems to get back to expected performance levels after exercising the different functions now - this didn't seem to be the case before. So fine by me to call it good.

chunga668 · 12-17-2010, 05:26 AM

I'd like to add my thanks to the growing list of fan club members.
I'll be away for the weekend and will check out the updates when I get back.
I will also be donating at that time!

chunga668 · 12-23-2010, 08:08 PM

Yeah, the latest updates seem to work great.
Even faster than before.
Thanks again for the speedy work!
I encourage everyone to donate if you haven't already.

kovidgoyal · 12-23-2010, 08:45 PM

You're welcome

12-15-2010, 02:14 PM	#17
kovidgoyal creator of calibre Posts: 45,400 Karma: 27756918 Join Date: Oct 2006 Location: Mumbai, India Device: Various	@ldolse: The fix for the mem leak in the edit metadata dialog has already been committed. Are you still seeing the memory usage of calibre climb as time passes (after adding a book)? Last edited by kovidgoyal; 12-15-2010 at 02:29 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Dramatically improved start-up time!	itimpi	Calibre	5	11-29-2010 11:03 AM
Faster or Slower?	Andrew Kaufman	General Discussions	18	07-23-2010 07:10 PM
Is it slower than before? or just me?	fred_yu_job	Kobo Reader	8	07-03-2010 09:32 AM
Red slower than Silver?	FligMupple	Sony Reader	11	02-01-2009 12:01 AM
Why is the reader slower with Calibre books?	rnfuller	Calibre	4	10-23-2008 07:03 PM

12-15-2010, 05:00 PM	#21
ldolse Wizard Posts: 1,337 Karma: 123455 Join Date: Apr 2009 Location: Malaysia Device: PRS-650, iPhone	Will try in a few hours, laptop power supply died, need to get a replacement. I was adding mostly epub, but there was also a mobi file. Metadata was being pulled from the book's internal metadata when adding rather than the filename.

12-15-2010, 10:06 PM	#23
ldolse Wizard Posts: 1,337 Karma: 123455 Join Date: Apr 2009 Location: Malaysia Device: PRS-650, iPhone	Yes, I saw that upon further testing. It turned out that I was loading more than just epub and mobi before, there were also some lit and lrf books in the group that I'd randomly grabbed for testing. Will initializing the different input plugins and metadata readers for each format use up some additional memory? Calibre started up using 209 megs of memory, and after adding a number of books (12 different titles, various formats) it rose up to 247-249 megs of memory (depending on which test run). It seemed like the memory usage stabilized somewhat at that point, but then I was adding and deleting the same group of books. Different formats seem to use varying amounts of memory, I can do some more testing around that. Note deleting all the books which caused the increase in memory didn't seem to free up any of that memory. Probably not a big deal, as that's not a wide use case. I'll also let it run overnight again - before I was seeing the slowness after letting it sit open for several hours.

12-15-2010, 10:18 PM	#24
kovidgoyal creator of calibre Posts: 45,400 Karma: 27756918 Join Date: Oct 2006 Location: Mumbai, India Device: Various	If you still get the slowdown tomorrow, then I'd be concerned. If you do, report the memory usage.

12-16-2010, 03:14 PM	#25
ldolse Wizard Posts: 1,337 Karma: 123455 Join Date: Apr 2009 Location: Malaysia Device: PRS-650, iPhone	Seems like it's better today, though it was quite slow when I initially accessed it. I think one of the things that was contributing to the perception of slowness is that Calibre gets paged out of 'real' memory to Virtual Memory. It takes a long time to get everything back into real memory, and during that period the UI is extremely sluggish for a good 10-15 seconds. After that adding books is quite slow for the first instance of each input format - the first LRF book I added took 3.6 seconds. Each additional format was similarly slow, but once all the different input plugin code had been 'exercised' everything was back to the speeds I was seeing the yesterday. I don't recall previous releases being so slow after leaving the program alone for a while. Did something change with the way things are paged in and out of virtual memory? Memory use was 154 megs, and jumped to 209 after using it a bit, so I don't think any leaks are the culprit.

12-16-2010, 03:23 PM	#26
kovidgoyal creator of calibre Posts: 45,400 Karma: 27756918 Join Date: Oct 2006 Location: Mumbai, India Device: Various	I can't think of anything that could cause more paging in newer releases. That said, in fixing memory leaks I have changed the way object references are stored/garbage collected in various parts of the calibre code and that might be causing this, though I don't know enough about the internals of python's garbage collection/windows memory management to speak with any confidence. As long as the speed recovers after the initial add of a format, I'm not going to spend more time on this. I will note that I don't notice any post idling slowdown on my linux machine but then that has 6 GB of RAM and linux never pages anything unless it really has to.

12-16-2010, 03:27 PM	#27
ldolse Wizard Posts: 1,337 Karma: 123455 Join Date: Apr 2009 Location: Malaysia Device: PRS-650, iPhone	Yeah, it definitely seems to get back to expected performance levels after exercising the different functions now - this didn't seem to be the case before. So fine by me to call it good.

12-17-2010, 05:26 AM	#28
chunga668 Junior Member Posts: 9 Karma: 10 Join Date: May 2010 Device: Nook	I'd like to add my thanks to the growing list of fan club members. I'll be away for the weekend and will check out the updates when I get back. I will also be donating at that time!

12-23-2010, 08:08 PM	#29
chunga668 Junior Member Posts: 9 Karma: 10 Join Date: May 2010 Device: Nook	Yeah, the latest updates seem to work great. Even faster than before. Thanks again for the speedy work! I encourage everyone to donate if you haven't already.

12-23-2010, 08:45 PM	#30
kovidgoyal creator of calibre Posts: 45,400 Karma: 27756918 Join Date: Oct 2006 Location: Mumbai, India Device: Various	You're welcome