Out of memory error in the latest version

Giuseppe Chillem · 06-11-2011, 04:40 PM

Hello,
I was renaming books. I have renamed about 100 of them and then an error popped up:

calibre, version 0.8.5
ERROR: Unhandled exception: <b>MemoryError</b>

ut of memory

Traceback (most recent call last):
File "site-packages\calibre\gui2\library\models.py", line 901, in setData
File "site-packages\calibre\library\database2.py", line 1784, in set
File "site-packages\calibre\library\database2.py", line 2083, in set_title
File "site-packages\calibre\library\database2.py", line 611, in set_path
File "shutil.py", line 51, in copyfileobj
File "tempfile.py", line 595, in write
MemoryError: out of memory

kovidgoyal · 06-11-2011, 04:59 PM

Were you renaming a book with a particularly large file?

Giuseppe Chillem · 06-11-2011, 05:04 PM

Yes, the book wich crashed calibre was 84,6 MB in size.

kovidgoyal · 06-12-2011, 10:37 AM

And how much RAM do you have on your computer?

Giuseppe Chillem · 06-12-2011, 11:00 AM

It is a 64Bit machine Dual Processor 8GB

kovidgoyal · 06-12-2011, 12:01 PM

That's weird. I cannot replicate this with a 90MB file on Win 7. And that traceback is indicating an out of memory error while copying from disk to memory, which cannot happen for only an 80MB file when you have several GB of RAM.

I'm at a loss.

kovidgoyal · 06-12-2011, 03:22 PM

I have reduced the max size stored in memory during the file copy, see if that makes a difference in the next release.

kovidgoyal · 06-12-2011, 03:31 PM

I think I know what's going on...stay tuned.

kovidgoyal · 06-12-2011, 04:07 PM

Quote:

Originally Posted by kovidgoyal

I think I know what's going on...stay tuned.

I was wrong, I still dont know whats going on

BradV · 06-22-2011, 08:26 AM

I am running into something similar. After downloading metadata for several books (as few as 6) Windows tells me that I need to close Calibre because I am running out of memory.

I watched the working set memory in TaskMan go up for the calibre.exe process each time I edited the metadata for a book (40-30mb each time). It seems to be more dramatic if I edit books rapidly, one after another. Could there be a memory leak here?

System Config
Windows 7 Enterprise (x86)
2GB RAM
Calibre 0.8.6

I was hoping the debug log would gather more information.

calibre Debug log
calibre 0.8.6
Windows-7-6.1.7600
Windows
('Windows', '7', '6.1.7600')
Python 2.7.1
Windows: ('7', '6.1.7600', '', 'Multiprocessor Free')
Starting up...
Started up in 18.0480000973

Thanks,
BradV

chaley · 06-22-2011, 09:29 AM

I tried an experiment using Win7 Pro/32, 4GB, calibre 0.8.6, 2000-book test library on C:, system on C:, calibre on C:. Memory numbers are what windows reports as "Working set".

Mem at start: 104MB.
Open edit metadata/single (default window layout): 113MB
Hit next button 20 times (commits the dialog): 128MB
Close edit metadata: 128MB
E/M a book and hit escape 20 times: 132MB
E/M a book, hit escape, hit down-arrow 20 times: 132MB
Select 20 books, EM Bulk, press OK: 132MB

At the end of this, Peak Working Set was 161MB.

There was a report that DEP (Data Execution Protection) made a difference. Mine is off. I turned it on and ran a few experiments, and saw no significant difference in memory use. It feels subjectively slightly faster without DEP, but that could be my imagination.

Bottom line: I see no evidence of a significant memory leak. Are you doing something very different from what I did?

BradV · 06-22-2011, 04:31 PM

Thanks for trying this Chaley. It looks like you were seeing a small leak as well. You went from 104mb to 132mb of working set. I think the issue is related to the number of books in my library. I was able to repro the issue on a second computer (using the same database).

Calibre 0.8.6
9268 book library on E:
OS on C:, Calibre on C:
Win7 Enterprise x86 (32-bit), 2gb RAM

Numbers are the “Working Set (Memory)” as reported by Task Manager

Repro Steps
Start Calibre (229,260k)
Select book, Edit Metadata (245,132k)
Download Metadata (257,778k)
Select 1st entry, Next
Select current cover (256,320k)
Click OK (285,672k)

Select book, edit metadata (296,580k)
Download metadata (305,888k)
Select 1st entry, Next
Select current cover (306,004k)
Click OK (336,568k)

That is over 100mb of working set memory by editing (downloading metadata for) 2 books.
When I attempted to repro the issue on a smaller book library (117 entries) I did not see anything this dramatic, but I did see a small amount of Working Set (Memory) increase with each edit.

Thanks,
BradV

chaley · 06-22-2011, 04:58 PM

Quote:

Originally Posted by BradV

Thanks for trying this Chaley. It looks like you were seeing a small leak as well.

Not convinced. In a GC language like python, it is very difficult to see the real usage. There might be free memory in python that windows thinks is used. I noted that sometimes if I left calibre alone for a few minutes, memory usage spontaneously reduced.

Quote:

You went from 104mb to 132mb of working set.

True. However, that is nowhere near the 40MB per book you were reporting, and it remained quite constant after the initial kick.

Quote:

I think the issue is related to the number of books in my library. I was able to repro the issue on a second computer (using the same database).

I tried it on a 20-book library, a 2000-book library, and a 4000-book library. The initial memory use varied, but the increases did not.

Quote:

Calibre 0.8.6
9268 book library on E:
OS on C:, Calibre on C:
Win7 Enterprise x86 (32-bit), 2gb RAM

Numbers are the “Working Set (Memory)” as reported by Task Manager

Repro Steps
Start Calibre (229,260k)
Select book, Edit Metadata (245,132k)
Download Metadata (257,778k)
Select 1st entry, Next
Select current cover (256,320k)
Click OK (285,672k)

Is E: a NAS or a second local drive?

There is clearly something different with your edit metadata. In every one of my tests on my three librarys, the maximum increase was 4 MB, and was often zero. You are seeing 16MB.

Are you using the default 2-tab edit metadata window? I don't think there is any difference, but one never knows. Are your covers particularly large (mine are all 500x500 or smaller)? If so, then memory must be allocated to display them, and this isn't an error.

We have another difference. I never use download metadata. and my tests didn't use it. There could be a leak in there, but that experiment will need to wait for another developer who uses it.

I also didn't change covers. which might also account for it.

BradV · 06-22-2011, 09:00 PM

Yes, I am using the default Edit Metadata window.
Some covers are large, some are small (less than 200x300).
The drive is a USB hard drive (local).

I will file a bug. Thanks!

BradV

kovidgoyal · 06-22-2011, 09:08 PM

No point filing a bug, without being able to replicate your memory leak, it is impossible to fix. I tried it on my Win 7 VM and after the initial jump of ~30MB the first time the dialog is opened and metadata downloaded, subsequent clicking of next or OK and opening again with/without metadata download or editing covers shows no change larger than a couple of MB in memory consumption.

06-11-2011, 04:40 PM	#1
Giuseppe Chillem Groupie Posts: 178 Karma: 134 Join Date: May 2010 Device: IREX DR1000	Out of memory error in the latest version Hello, I was renaming books. I have renamed about 100 of them and then an error popped up: calibre, version 0.8.5 ERROR: Unhandled exception: <b>MemoryError</b>ut of memory Traceback (most recent call last): File "site-packages\calibre\gui2\library\models.py", line 901, in setData File "site-packages\calibre\library\database2.py", line 1784, in set File "site-packages\calibre\library\database2.py", line 2083, in set_title File "site-packages\calibre\library\database2.py", line 611, in set_path File "shutil.py", line 51, in copyfileobj File "tempfile.py", line 595, in write MemoryError: out of memory

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
How do I update Calibre to the latest version?	nsivakr	Calibre	18	02-22-2017 11:07 AM
Kindle 3G 6" Latest gen. memory is 256MB or 2GB?	dagero	Kindle Developer's Corner	8	11-11-2010 04:20 PM
is there virus in the latest version	ichthyosaur	Calibre	1	01-19-2010 03:58 PM
latest version for Ubuntu	Lago	Calibre	7	01-15-2010 01:54 PM
Guardian Recipe messed up in latest version??	pars_andy	Calibre	6	11-14-2009 03:50 PM

06-11-2011, 04:59 PM	#2
kovidgoyal creator of calibre Posts: 43,835 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Were you renaming a book with a particularly large file?

06-11-2011, 05:04 PM	#3
Giuseppe Chillem Groupie Posts: 178 Karma: 134 Join Date: May 2010 Device: IREX DR1000	Yes, the book wich crashed calibre was 84,6 MB in size.

06-12-2011, 10:37 AM	#4
kovidgoyal creator of calibre Posts: 43,835 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	And how much RAM do you have on your computer?

06-12-2011, 11:00 AM	#5
Giuseppe Chillem Groupie Posts: 178 Karma: 134 Join Date: May 2010 Device: IREX DR1000	It is a 64Bit machine Dual Processor 8GB

06-12-2011, 12:01 PM	#6
kovidgoyal creator of calibre Posts: 43,835 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	That's weird. I cannot replicate this with a 90MB file on Win 7. And that traceback is indicating an out of memory error while copying from disk to memory, which cannot happen for only an 80MB file when you have several GB of RAM. I'm at a loss.

06-12-2011, 03:22 PM	#7
kovidgoyal creator of calibre Posts: 43,835 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	I have reduced the max size stored in memory during the file copy, see if that makes a difference in the next release.

06-12-2011, 03:31 PM	#8
kovidgoyal creator of calibre Posts: 43,835 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	I think I know what's going on...stay tuned.

06-22-2011, 08:26 AM	#10
BradV Member Posts: 10 Karma: 10 Join Date: Jun 2011 Device: Kindle3	I am running into something similar. After downloading metadata for several books (as few as 6) Windows tells me that I need to close Calibre because I am running out of memory. I watched the working set memory in TaskMan go up for the calibre.exe process each time I edited the metadata for a book (40-30mb each time). It seems to be more dramatic if I edit books rapidly, one after another. Could there be a memory leak here? System Config Windows 7 Enterprise (x86) 2GB RAM Calibre 0.8.6 I was hoping the debug log would gather more information. calibre Debug log calibre 0.8.6 Windows-7-6.1.7600 Windows ('Windows', '7', '6.1.7600') Python 2.7.1 Windows: ('7', '6.1.7600', '', 'Multiprocessor Free') Starting up... Started up in 18.0480000973 Thanks, BradV

06-22-2011, 09:29 AM	#11
chaley Grand Sorcerer Posts: 11,731 Karma: 6690881 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	I tried an experiment using Win7 Pro/32, 4GB, calibre 0.8.6, 2000-book test library on C:, system on C:, calibre on C:. Memory numbers are what windows reports as "Working set". Mem at start: 104MB. Open edit metadata/single (default window layout): 113MB Hit next button 20 times (commits the dialog): 128MB Close edit metadata: 128MB E/M a book and hit escape 20 times: 132MB E/M a book, hit escape, hit down-arrow 20 times: 132MB Select 20 books, EM Bulk, press OK: 132MB At the end of this, Peak Working Set was 161MB. There was a report that DEP (Data Execution Protection) made a difference. Mine is off. I turned it on and ran a few experiments, and saw no significant difference in memory use. It feels subjectively slightly faster without DEP, but that could be my imagination. Bottom line: I see no evidence of a significant memory leak. Are you doing something very different from what I did?

06-22-2011, 04:31 PM	#12
BradV Member Posts: 10 Karma: 10 Join Date: Jun 2011 Device: Kindle3	Thanks for trying this Chaley. It looks like you were seeing a small leak as well. You went from 104mb to 132mb of working set. I think the issue is related to the number of books in my library. I was able to repro the issue on a second computer (using the same database). Calibre 0.8.6 9268 book library on E: OS on C:, Calibre on C: Win7 Enterprise x86 (32-bit), 2gb RAM Numbers are the “Working Set (Memory)” as reported by Task Manager Repro Steps Start Calibre (229,260k) Select book, Edit Metadata (245,132k) Download Metadata (257,778k) Select 1st entry, Next Select current cover (256,320k) Click OK (285,672k) Select book, edit metadata (296,580k) Download metadata (305,888k) Select 1st entry, Next Select current cover (306,004k) Click OK (336,568k) That is over 100mb of working set memory by editing (downloading metadata for) 2 books. When I attempted to repro the issue on a smaller book library (117 entries) I did not see anything this dramatic, but I did see a small amount of Working Set (Memory) increase with each edit. Thanks, BradV

Advert

Advert

06-22-2011, 09:00 PM	#14
BradV Member Posts: 10 Karma: 10 Join Date: Jun 2011 Device: Kindle3	Yes, I am using the default Edit Metadata window. Some covers are large, some are small (less than 200x300). The drive is a USB hard drive (local). I will file a bug. Thanks! BradV

06-22-2011, 09:08 PM	#15
kovidgoyal creator of calibre Posts: 43,835 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	No point filing a bug, without being able to replicate your memory leak, it is impossible to fix. I tried it on my Win 7 VM and after the initial jump of ~30MB the first time the dialog is opened and metadata downloaded, subsequent clicking of next or OK and opening again with/without metadata download or editing covers shows no change larger than a couple of MB in memory consumption.