Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 06-11-2011, 04:40 PM   #1
Giuseppe Chillem
Zealot
Giuseppe Chillem doesn't litterGiuseppe Chillem doesn't litter
 
Giuseppe Chillem's Avatar
 
Posts: 135
Karma: 134
Join Date: May 2010
Device: IREX DR1000
Out of memory error in the latest version

Hello,
I was renaming books. I have renamed about 100 of them and then an error popped up:

calibre, version 0.8.5
ERROR: Unhandled exception: <b>MemoryError</b>ut of memory

Traceback (most recent call last):
File "site-packages\calibre\gui2\library\models.py", line 901, in setData
File "site-packages\calibre\library\database2.py", line 1784, in set
File "site-packages\calibre\library\database2.py", line 2083, in set_title
File "site-packages\calibre\library\database2.py", line 611, in set_path
File "shutil.py", line 51, in copyfileobj
File "tempfile.py", line 595, in write
MemoryError: out of memory
Giuseppe Chillem is offline   Reply With Quote
Old 06-11-2011, 04:59 PM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,103
Karma: 5101571
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Were you renaming a book with a particularly large file?
kovidgoyal is offline   Reply With Quote
Old 06-11-2011, 05:04 PM   #3
Giuseppe Chillem
Zealot
Giuseppe Chillem doesn't litterGiuseppe Chillem doesn't litter
 
Giuseppe Chillem's Avatar
 
Posts: 135
Karma: 134
Join Date: May 2010
Device: IREX DR1000
Yes, the book wich crashed calibre was 84,6 MB in size.
Giuseppe Chillem is offline   Reply With Quote
Old 06-12-2011, 10:37 AM   #4
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,103
Karma: 5101571
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
And how much RAM do you have on your computer?
kovidgoyal is offline   Reply With Quote
Old 06-12-2011, 11:00 AM   #5
Giuseppe Chillem
Zealot
Giuseppe Chillem doesn't litterGiuseppe Chillem doesn't litter
 
Giuseppe Chillem's Avatar
 
Posts: 135
Karma: 134
Join Date: May 2010
Device: IREX DR1000
It is a 64Bit machine Dual Processor 8GB
Giuseppe Chillem is offline   Reply With Quote
Old 06-12-2011, 12:01 PM   #6
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,103
Karma: 5101571
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
That's weird. I cannot replicate this with a 90MB file on Win 7. And that traceback is indicating an out of memory error while copying from disk to memory, which cannot happen for only an 80MB file when you have several GB of RAM.

I'm at a loss.
kovidgoyal is offline   Reply With Quote
Old 06-12-2011, 03:22 PM   #7
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,103
Karma: 5101571
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
I have reduced the max size stored in memory during the file copy, see if that makes a difference in the next release.
kovidgoyal is offline   Reply With Quote
Old 06-12-2011, 03:31 PM   #8
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,103
Karma: 5101571
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
I think I know what's going on...stay tuned.
kovidgoyal is offline   Reply With Quote
Old 06-12-2011, 04:07 PM   #9
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,103
Karma: 5101571
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Quote:
Originally Posted by kovidgoyal View Post
I think I know what's going on...stay tuned.
I was wrong, I still dont know whats going on
kovidgoyal is offline   Reply With Quote
Old 06-22-2011, 08:26 AM   #10
BradV
Member
BradV began at the beginning.
 
Posts: 10
Karma: 10
Join Date: Jun 2011
Device: Kindle3
I am running into something similar. After downloading metadata for several books (as few as 6) Windows tells me that I need to close Calibre because I am running out of memory.

I watched the working set memory in TaskMan go up for the calibre.exe process each time I edited the metadata for a book (40-30mb each time). It seems to be more dramatic if I edit books rapidly, one after another. Could there be a memory leak here?

System Config
Windows 7 Enterprise (x86)
2GB RAM
Calibre 0.8.6

I was hoping the debug log would gather more information.

calibre Debug log
calibre 0.8.6
Windows-7-6.1.7600
Windows
('Windows', '7', '6.1.7600')
Python 2.7.1
Windows: ('7', '6.1.7600', '', 'Multiprocessor Free')
Starting up...
Started up in 18.0480000973

Thanks,
BradV
BradV is offline   Reply With Quote
Old 06-22-2011, 09:29 AM   #11
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 5,619
Karma: 857074
Join Date: Jan 2010
Location: France
Device: Many android devices
I tried an experiment using Win7 Pro/32, 4GB, calibre 0.8.6, 2000-book test library on C:, system on C:, calibre on C:. Memory numbers are what windows reports as "Working set".

Mem at start: 104MB.
Open edit metadata/single (default window layout): 113MB
Hit next button 20 times (commits the dialog): 128MB
Close edit metadata: 128MB
E/M a book and hit escape 20 times: 132MB
E/M a book, hit escape, hit down-arrow 20 times: 132MB
Select 20 books, EM Bulk, press OK: 132MB

At the end of this, Peak Working Set was 161MB.

There was a report that DEP (Data Execution Protection) made a difference. Mine is off. I turned it on and ran a few experiments, and saw no significant difference in memory use. It feels subjectively slightly faster without DEP, but that could be my imagination.

Bottom line: I see no evidence of a significant memory leak. Are you doing something very different from what I did?
chaley is offline   Reply With Quote
Old 06-22-2011, 04:31 PM   #12
BradV
Member
BradV began at the beginning.
 
Posts: 10
Karma: 10
Join Date: Jun 2011
Device: Kindle3
Thanks for trying this Chaley. It looks like you were seeing a small leak as well. You went from 104mb to 132mb of working set. I think the issue is related to the number of books in my library. I was able to repro the issue on a second computer (using the same database).

Calibre 0.8.6
9268 book library on E:
OS on C:, Calibre on C:
Win7 Enterprise x86 (32-bit), 2gb RAM

Numbers are the “Working Set (Memory)” as reported by Task Manager

Repro Steps
Start Calibre (229,260k)
Select book, Edit Metadata (245,132k)
Download Metadata (257,778k)
Select 1st entry, Next
Select current cover (256,320k)
Click OK (285,672k)

Select book, edit metadata (296,580k)
Download metadata (305,888k)
Select 1st entry, Next
Select current cover (306,004k)
Click OK (336,568k)

That is over 100mb of working set memory by editing (downloading metadata for) 2 books.
When I attempted to repro the issue on a smaller book library (117 entries) I did not see anything this dramatic, but I did see a small amount of Working Set (Memory) increase with each edit.

Thanks,
BradV
BradV is offline   Reply With Quote
Old 06-22-2011, 04:58 PM   #13
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 5,619
Karma: 857074
Join Date: Jan 2010
Location: France
Device: Many android devices
Quote:
Originally Posted by BradV View Post
Thanks for trying this Chaley. It looks like you were seeing a small leak as well.
Not convinced. In a GC language like python, it is very difficult to see the real usage. There might be free memory in python that windows thinks is used. I noted that sometimes if I left calibre alone for a few minutes, memory usage spontaneously reduced.
Quote:
You went from 104mb to 132mb of working set.
True. However, that is nowhere near the 40MB per book you were reporting, and it remained quite constant after the initial kick.
Quote:
I think the issue is related to the number of books in my library. I was able to repro the issue on a second computer (using the same database).
I tried it on a 20-book library, a 2000-book library, and a 4000-book library. The initial memory use varied, but the increases did not.
Quote:
Calibre 0.8.6
9268 book library on E:
OS on C:, Calibre on C:
Win7 Enterprise x86 (32-bit), 2gb RAM

Numbers are the “Working Set (Memory)” as reported by Task Manager

Repro Steps
Start Calibre (229,260k)
Select book, Edit Metadata (245,132k)
Download Metadata (257,778k)
Select 1st entry, Next
Select current cover (256,320k)
Click OK (285,672k)
Is E: a NAS or a second local drive?

There is clearly something different with your edit metadata. In every one of my tests on my three librarys, the maximum increase was 4 MB, and was often zero. You are seeing 16MB.

Are you using the default 2-tab edit metadata window? I don't think there is any difference, but one never knows. Are your covers particularly large (mine are all 500x500 or smaller)? If so, then memory must be allocated to display them, and this isn't an error.

We have another difference. I never use download metadata. and my tests didn't use it. There could be a leak in there, but that experiment will need to wait for another developer who uses it.

I also didn't change covers. which might also account for it.
chaley is offline   Reply With Quote
Old 06-22-2011, 09:00 PM   #14
BradV
Member
BradV began at the beginning.
 
Posts: 10
Karma: 10
Join Date: Jun 2011
Device: Kindle3
Yes, I am using the default Edit Metadata window.
Some covers are large, some are small (less than 200x300).
The drive is a USB hard drive (local).

I will file a bug. Thanks!

BradV
BradV is offline   Reply With Quote
Old 06-22-2011, 09:08 PM   #15
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,103
Karma: 5101571
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
No point filing a bug, without being able to replicate your memory leak, it is impossible to fix. I tried it on my Win 7 VM and after the initial jump of ~30MB the first time the dialog is opened and metadata downloaded, subsequent clicking of next or OK and opening again with/without metadata download or editing covers shows no change larger than a couple of MB in memory consumption.
kovidgoyal is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
How do I update Calibre to the latest version? nsivakr Calibre 12 04-25-2011 06:10 AM
Kindle 3G 6" Latest gen. memory is 256MB or 2GB? dagero Kindle Developer's Corner 8 11-11-2010 04:20 PM
is there virus in the latest version ichthyosaur Calibre 1 01-19-2010 03:58 PM
latest version for Ubuntu Lago Calibre 7 01-15-2010 01:54 PM
Guardian Recipe messed up in latest version?? pars_andy Calibre 6 11-14-2009 03:50 PM


All times are GMT -4. The time now is 04:59 AM.


MobileRead.com is a privately owned, operated and funded community.