02-16-2011, 12:27 AM | #1 |
Enthusiast
Posts: 38
Karma: 10
Join Date: Jan 2011
Device: none
|
0.7.45 (and earlier versions) conversion crash
0.7.45 on WinXP dual core system
I am attempting to batch convert about 6000 epubs to mobi using default conversion options. I get a crash about once in every 20-50 books. It is not a new problem. I have observed this problem on several Calibre versions over the last 8 weeks. If I leave the option "Maximum number of waiting worker process" at the default 6 then the crash causes Calibre (and all jobs) to terminate (and occasionally blue screen's XP). If I reduce this option to 2, then the active job crashes with an error but Calibre continues to operate and starts the next job. I have a hunch that the cause of the problem is a memory leak. The time required to convert an epub to mobi can vary from a minute to several hours. I observe with Windows Task Manager that memory usage by calibre-parallel.exe increases constantly while the thread is running. For example, it starts at about 60 MB and 40 minutes later is consuming 300 MB. My hunch is that epubs with exceptionally long conversion times exceed some memory limit in Windows and crash. For example, I see that a job that was 107 minutes into a conversion eventually crashed. I can provide logs and sample epubs on request but I suspect this will be very easy to reproduce. Just find an epub that is slow to convert and watch what happens with Windows Task Manager. Thanks for an amazing product! |
02-16-2011, 11:48 AM | #2 |
creator of calibre
Posts: 43,866
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Conversion is not designed to minimize memory usage, it's designed to be as flexible and powerful as possible. Since each conversion runs in it's own process and the process is killed after the conversion completes, the only way that memory consumption can cause a crash is if the memory consumption of a single conversion is larger than the amount of RAM + swap on your machine. I find that highly unlikely.
Given that your windows is blue screening, which should never happen with a conversion (conversion is a user space process), I would suspect you have hardware issues, most likely something breaks under heavy load. That said, if you have an epub that consistently causes a crash every time you try to convert it, open a ticket and attach it and the conversion logs. |
02-16-2011, 12:04 PM | #3 |
Enthusiast
Posts: 38
Karma: 10
Join Date: Jan 2011
Device: none
|
Thanks.
After 6 years of heavy use with diverse applications I am pretty sure I do not have hardware problems. And I have not seen a blue screen for 8 weeks since I dropped max process to 2. I have seen repeated crashes, say one every 2 hours, but frequency is highly dependent on epub content. I will find an epub that repeatably crashes and send to you. |
02-16-2011, 02:07 PM | #4 | |
Well trained by Cats
Posts: 29,812
Karma: 54830978
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
|
Quote:
I was getting (very) occasional BSOD's , then it became common. I started Memtest 86+ and did not have to wait long Red item started showing.. 1 Stick (of a matched pair ) had gone belly up. Memtest86 comes standard on a Ubuntu Live (bootable) CD as well as a standalone ISO (to burn your own) |
|
02-16-2011, 04:52 PM | #5 |
Enthusiast
Posts: 38
Karma: 10
Join Date: Jan 2011
Device: none
|
Is there anyone out there that has successfully converted several hundred epubs to mobi via batch convert in the last month or two?
I think I am observing a repeatable bug but will revise my view if others are not seeing it. |
02-16-2011, 04:58 PM | #6 |
Connoisseur
Posts: 67
Karma: 40
Join Date: Aug 2010
Device: iPad, Kindle Paperwhite
|
I did convert 400 epubs to mobi about one month ago, probably running 0.7.35 or below.
I didn't found any problem other than for some unknown reason my converted mobis didn't have any tag applied to them. |
02-16-2011, 06:36 PM | #7 |
Wizard
Posts: 4,552
Karma: 950151
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
|
I did a run converting several hundred books using the 0.7.45 release last weekend.
|
02-16-2011, 10:12 PM | #8 | |||
US Navy, Retired
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
|
Quote:
Everything you describe except the crashes are normal. The ePubs that take forever inevitably have their origin as a MS Word doc that was saved as html without being filtered. We are aware of these and they don't have a history of crashing, just taking forever to complete. Some have reported 24 hours, the longest I've seen is 6-7 hours. Quote:
Quote:
I have the following settings Waiting worker processes : 4 Job Priority : Low Calibre library exempted from Anti-virus scanning. Calibre configuration folder exempted from A/V scanning. I believe the problems you're experiencing are unique to your system. Look at what other programs/processes are running. Did you add anything new to the startup in the last 3 months? Sorry I couldn't be of more help. |
|||
02-17-2011, 01:51 PM | #9 |
Enthusiast
Posts: 38
Karma: 10
Join Date: Jan 2011
Device: none
|
Thanks to everyone for the help.
I've just started a batch job of "problematic" epubs on a different system (Vista). This will eliminate any possibility of hardware or system problems. I'll be back with the results shortly. |
02-17-2011, 02:11 PM | #10 |
Enthusiast
Posts: 38
Karma: 10
Join Date: Jan 2011
Device: none
|
Well that did not take long...
I adjusted max waiting processes to 4 as per dwanthy's recommendation. About 5 minutes into the conversion my Vista system did a hard lock up. By this I mean both Calibre and Vista stopped responding. Even ctl-alt-del is ignored. I had to cycle power to reboot. This result is consistent with my blue screen observations on my XP system whenever I set max waiting processes greater than 2. So in summary I am seeing a hard crash on 2 different hardware systems with 2 different operating systems. Please advise next step. I assume I have no log to send because of hard crash, but I can send epub if someone provides destination. |
02-17-2011, 02:14 PM | #11 |
creator of calibre
Posts: 43,866
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Again, if you can isolate a single epub that consistently (which means every time you try to convert it) causes a crash, open a ticket and attach the epub.
|
02-17-2011, 03:52 PM | #12 |
Enthusiast
Posts: 38
Karma: 10
Join Date: Jan 2011
Device: none
|
ticket opened with attached epub
thanks |
02-17-2011, 10:56 PM | #13 | ||
US Navy, Retired
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
|
Quote:
Quote:
That is why I have my machine setup as follows. Calibre - Preferences - Behavior - Job Priority : Low Calibre library exempted from Anti-virus scanning. Calibre configuration folder exempted from A/V scanning. With these settings my machine, like yours, went from having all resources go to calibre to being a completely responsive machine able to handle multiple tasks while converting those 300 epubs that I experimented with to help resolve this problem you're having. |
||
02-18-2011, 12:10 PM | #14 |
Enthusiast
Posts: 38
Karma: 10
Join Date: Jan 2011
Device: none
|
Thanks for the tips dwanthy. Kovid reported in the bug tracker that the epub that crashed both of my systems was "pathological". He hopes to add a filter to screen similar bad epubs.
|
02-19-2011, 10:28 AM | #15 |
Enthusiast
Posts: 38
Karma: 10
Join Date: Jan 2011
Device: none
|
One final comment to close out this thread.
I have just completed converting 3 large (about 800) batches of epubs. Calibre worked perfectly. The lesson for anyone reading this thread is that if Calibre or your system crashes during a conversion then it probably means you have a defective epub. Exclude (or delete) that epub and restart your batch conversion. |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Super-slow conversion, system crash? | Seanette | Calibre | 2 | 11-14-2010 07:35 AM |
PRS-505 Earlier versions of Sony Reader Library? | captinlizi | Sony Reader | 5 | 10-02-2010 04:38 PM |
6.13 Crash during conversion | jquest | Calibre | 1 | 09-21-2009 05:02 PM |
Critical Vulnerability in Adobe Reader 9, Acrobat 9 and earlier versions. | Patricia | 1 | 02-22-2009 10:14 AM | |
any2xxx, lrf2xxx not working in calibre 0.4.125 (and earlier versions) | Oletros | Calibre | 2 | 01-04-2009 12:54 PM |