Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Conversion

Notices

Reply
 
Thread Tools Search this Thread
Old 02-16-2011, 12:27 AM   #1
comox
Enthusiast
comox began at the beginning.
 
Posts: 38
Karma: 10
Join Date: Jan 2011
Device: none
0.7.45 (and earlier versions) conversion crash

0.7.45 on WinXP dual core system

I am attempting to batch convert about 6000 epubs to mobi using default conversion options.

I get a crash about once in every 20-50 books. It is not a new problem. I have observed this problem on several Calibre versions over the last 8 weeks.

If I leave the option "Maximum number of waiting worker process" at the default 6 then the crash causes Calibre (and all jobs) to terminate (and occasionally blue screen's XP). If I reduce this option to 2, then the active job crashes with an error but Calibre continues to operate and starts the next job.

I have a hunch that the cause of the problem is a memory leak. The time required to convert an epub to mobi can vary from a minute to several hours. I observe with Windows Task Manager that memory usage by calibre-parallel.exe increases constantly while the thread is running. For example, it starts at about 60 MB and 40 minutes later is consuming 300 MB. My hunch is that epubs with exceptionally long conversion times exceed some memory limit in Windows and crash. For example, I see that a job that was 107 minutes into a conversion eventually crashed.

I can provide logs and sample epubs on request but I suspect this will be very easy to reproduce. Just find an epub that is slow to convert and watch what happens with Windows Task Manager.

Thanks for an amazing product!
comox is offline   Reply With Quote
Old 02-16-2011, 11:48 AM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,744
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Conversion is not designed to minimize memory usage, it's designed to be as flexible and powerful as possible. Since each conversion runs in it's own process and the process is killed after the conversion completes, the only way that memory consumption can cause a crash is if the memory consumption of a single conversion is larger than the amount of RAM + swap on your machine. I find that highly unlikely.

Given that your windows is blue screening, which should never happen with a conversion (conversion is a user space process), I would suspect you have hardware issues, most likely something breaks under heavy load.

That said, if you have an epub that consistently causes a crash every time you try to convert it, open a ticket and attach it and the conversion logs.
kovidgoyal is offline   Reply With Quote
Advert
Old 02-16-2011, 12:04 PM   #3
comox
Enthusiast
comox began at the beginning.
 
Posts: 38
Karma: 10
Join Date: Jan 2011
Device: none
Thanks.

After 6 years of heavy use with diverse applications I am pretty sure I do not have hardware problems. And I have not seen a blue screen for 8 weeks since I dropped max process to 2. I have seen repeated crashes, say one every 2 hours, but frequency is highly dependent on epub content.

I will find an epub that repeatably crashes and send to you.
comox is offline   Reply With Quote
Old 02-16-2011, 02:07 PM   #4
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 29,659
Karma: 54369090
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
Quote:
Originally Posted by comox View Post
Thanks.

After 6 years of heavy use with diverse applications I am pretty sure I do not have hardware problems. And I have not seen a blue screen for 8 weeks since I dropped max process to 2. I have seen repeated crashes, say one every 2 hours, but frequency is highly dependent on epub content.

I will find an epub that repeatably crashes and send to you.
If you have not run Memtest86+ , I highly recommend it.
I was getting (very) occasional BSOD's , then it became common. I started Memtest 86+ and did not have to wait long
Red item started showing.. 1 Stick (of a matched pair ) had gone belly up.
Memtest86 comes standard on a Ubuntu Live (bootable) CD as well as a standalone ISO (to burn your own)
theducks is offline   Reply With Quote
Old 02-16-2011, 04:52 PM   #5
comox
Enthusiast
comox began at the beginning.
 
Posts: 38
Karma: 10
Join Date: Jan 2011
Device: none
Is there anyone out there that has successfully converted several hundred epubs to mobi via batch convert in the last month or two?

I think I am observing a repeatable bug but will revise my view if others are not seeing it.
comox is offline   Reply With Quote
Advert
Old 02-16-2011, 04:58 PM   #6
Coleccionista
Connoisseur
Coleccionista began at the beginning.
 
Posts: 67
Karma: 40
Join Date: Aug 2010
Device: iPad, Kindle Paperwhite
I did convert 400 epubs to mobi about one month ago, probably running 0.7.35 or below.

I didn't found any problem other than for some unknown reason my converted mobis didn't have any tag applied to them.
Coleccionista is offline   Reply With Quote
Old 02-16-2011, 06:36 PM   #7
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,552
Karma: 950151
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
I did a run converting several hundred books using the 0.7.45 release last weekend.
itimpi is offline   Reply With Quote
Old 02-16-2011, 10:12 PM   #8
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Quote:
Originally Posted by comox View Post
I have a hunch that the cause of the problem is a memory leak. The time required to convert an epub to mobi can vary from a minute to several hours. I observe with Windows Task Manager that memory usage by calibre-parallel.exe increases constantly while the thread is running.
It is definitely not a memory leak.

Everything you describe except the crashes are normal. The ePubs that take forever inevitably have their origin as a MS Word doc that was saved as html without being filtered. We are aware of these and they don't have a history of crashing, just taking forever to complete. Some have reported 24 hours, the longest I've seen is 6-7 hours.

Quote:
Originally Posted by comox View Post
I suspect this will be very easy to reproduce. Just find an epub that is slow to convert and watch what happens with Windows Task Manager
Increasing memory is normal for conversions the longer they take.

Quote:
Originally Posted by comox View Post
Is there anyone out there that has successfully converted several hundred epubs to mobi via batch convert in the last month or two?

I think I am observing a repeatable bug but will revise my view if others are not seeing it.
I am also running calibre 0.7.45 on Win XP dual core system. I just converted 300 ePubs to Mobi without incident.

I have the following settings

Waiting worker processes : 4
Job Priority : Low
Calibre library exempted from Anti-virus scanning.
Calibre configuration folder exempted from A/V scanning.

I believe the problems you're experiencing are unique to your system. Look at what other programs/processes are running. Did you add anything new to the startup in the last 3 months?

Sorry I couldn't be of more help.
DoctorOhh is offline   Reply With Quote
Old 02-17-2011, 01:51 PM   #9
comox
Enthusiast
comox began at the beginning.
 
Posts: 38
Karma: 10
Join Date: Jan 2011
Device: none
Thanks to everyone for the help.

I've just started a batch job of "problematic" epubs on a different system (Vista).

This will eliminate any possibility of hardware or system problems.

I'll be back with the results shortly.
comox is offline   Reply With Quote
Old 02-17-2011, 02:11 PM   #10
comox
Enthusiast
comox began at the beginning.
 
Posts: 38
Karma: 10
Join Date: Jan 2011
Device: none
Well that did not take long...

I adjusted max waiting processes to 4 as per dwanthy's recommendation.

About 5 minutes into the conversion my Vista system did a hard lock up. By this I mean both Calibre and Vista stopped responding. Even ctl-alt-del is ignored. I had to cycle power to reboot.

This result is consistent with my blue screen observations on my XP system whenever I set max waiting processes greater than 2.

So in summary I am seeing a hard crash on 2 different hardware systems with 2 different operating systems.

Please advise next step. I assume I have no log to send because of hard crash, but I can send epub if someone provides destination.
comox is offline   Reply With Quote
Old 02-17-2011, 02:14 PM   #11
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,744
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Again, if you can isolate a single epub that consistently (which means every time you try to convert it) causes a crash, open a ticket and attach the epub.
kovidgoyal is offline   Reply With Quote
Old 02-17-2011, 03:52 PM   #12
comox
Enthusiast
comox began at the beginning.
 
Posts: 38
Karma: 10
Join Date: Jan 2011
Device: none
ticket opened with attached epub
thanks
comox is offline   Reply With Quote
Old 02-17-2011, 10:56 PM   #13
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Quote:
Originally Posted by comox View Post
I adjusted max waiting processes to 4 as per dwanthy's recommendation.

About 5 minutes into the conversion my Vista system did a hard lock up. By this I mean both Calibre and Vista stopped responding. Even ctl-alt-del is ignored. I had to cycle power to reboot.
You set the processes to 4, not a suggestion but what I had my system set to. If you want to keep your system responsive while it is converting, and it was still converting, then you should try my other settings, specifically go to calibre Preferences - Behavior - Job priority and change it from Normal to Low. With this change your computer will remain responsive.

Quote:
Originally Posted by comox View Post
So in summary I am seeing a hard crash on 2 different hardware systems with 2 different operating systems.
So in summary you have a hard crash on one machine and a not unexpected result on the second. The second may be a bug that Kovid may or may not be able to address, but for machines like your Vista machine the setting for job priority is a solution.

That is why I have my machine setup as follows.

Calibre - Preferences - Behavior - Job Priority : Low
Calibre library exempted from Anti-virus scanning.
Calibre configuration folder exempted from A/V scanning.

With these settings my machine, like yours, went from having all resources go to calibre to being a completely responsive machine able to handle multiple tasks while converting those 300 epubs that I experimented with to help resolve this problem you're having.
DoctorOhh is offline   Reply With Quote
Old 02-18-2011, 12:10 PM   #14
comox
Enthusiast
comox began at the beginning.
 
Posts: 38
Karma: 10
Join Date: Jan 2011
Device: none
Thanks for the tips dwanthy. Kovid reported in the bug tracker that the epub that crashed both of my systems was "pathological". He hopes to add a filter to screen similar bad epubs.
comox is offline   Reply With Quote
Old 02-19-2011, 10:28 AM   #15
comox
Enthusiast
comox began at the beginning.
 
Posts: 38
Karma: 10
Join Date: Jan 2011
Device: none
One final comment to close out this thread.

I have just completed converting 3 large (about 800) batches of epubs. Calibre worked perfectly.

The lesson for anyone reading this thread is that if Calibre or your system crashes during a conversion then it probably means you have a defective epub. Exclude (or delete) that epub and restart your batch conversion.
comox is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Super-slow conversion, system crash? Seanette Calibre 2 11-14-2010 07:35 AM
PRS-505 Earlier versions of Sony Reader Library? captinlizi Sony Reader 5 10-02-2010 04:38 PM
6.13 Crash during conversion jquest Calibre 1 09-21-2009 05:02 PM
Critical Vulnerability in Adobe Reader 9, Acrobat 9 and earlier versions. Patricia PDF 1 02-22-2009 10:14 AM
any2xxx, lrf2xxx not working in calibre 0.4.125 (and earlier versions) Oletros Calibre 2 01-04-2009 12:54 PM


All times are GMT -4. The time now is 03:01 AM.


MobileRead.com is a privately owned, operated and funded community.