[Old Thread] Large Bulk Conversion

[penguinaka]

Has anyone else had the issue of calibre hanging when trying to bulk convert jobs large jobs like 18,000+

I found that i'm having to sort by author and then do like 3000 at a time.

something else thats irratating is that im getting a bunch of errors and this little error window stacks up and i will have to go through each one and write down the error into notpad individually so i can go back an work on those files

It would be nice if the jobs list was sortable by error or finished so it could be copied out of there... or all the errors could be spat out in a log file. Is there some such thing?

And any suggestion on what i can do to prevent it hanging on large jobs?

Intel 2 Quad q6600 @ 2.40ghz 6gb ram 64bit windows 7

so i'm pretty sure i have the computing power.

[Manichean]

Moderator Notice
Moved to correct forum.

What exactly do you mean by "hanging"? If you mean that Calibre takes a long time on a certain conversion, that is probably normal. Depending on the quality of the source, it can take quite some time to convert certain books.

[kovidgoyal]

Run calibre in debug mode and you will get a debug log.

[penguinaka]

hanging = calibre freezes up doesnt respond and after waiting 30 minutes its still in such a state that its unlikely ever to continue and actually start doing the job.

It hangs after it has read the job list but before it starts doing the actual job conversions... so the little job counter still shows as 0

[penguinaka]

@kovidgoyal ill try that after this next batch of 3k is done thanks...
so when running in debug mode and calibre freezes and u have to end task and restart where do u retrieve the bug file?

[Manichean]

If you're trying to add all 18.000 files at once, I'd expect that to take about six times as long as adding 3.000 books to the queue.
"Unlikely ever to continue" is a very vague and subjective statement, by the way. You'll have to remember that Calibre is written in Python, which is, despite its obvious benefits, not the most performant of programming languages. Doing certain things may take a little while.

As for the debug log, you'll get a message on startup telling you where the log will be saved. The log will also automatically be displayed once Calibre is terminated.

[penguinaka]

"Unlikely ever to continue" is also very vague and subjective statement, by the way lol. Are we talking 6 x 30min for a total of 3 hours before it begins to actually convert. When i ctrl-alt-del run task manager and check out processes it has it listed as "Not Responding"

Originally Posted by Manichean

As for the debug log, you'll get a message on startup telling you where the log will be saved. The log will also automatically be displayed once Calibre is terminated.

Thanks very helpful

cheers

[Manichean]

Originally Posted by penguinaka

Are we talking 6 x 30min for a total of 3 hours before it begins to actually convert. When i ctrl-alt-del run task manager and check out processes it has it listed as "Not Responding"

I wouldn't know. If it takes you 30 minutes to add 3.000 books to the conversion queue and start converting, then I'd guess 18.000 books would take about 3 hours, if not a little longer.
As far as I know, Windows polls tasks somewhat regularly and if they don't respond, they get marked as "not responding", which doesn't necessarily mean that they aren't working. I've had programs doing calculations run for quite some time with a status "not responding" and finish their job. Like I said, Python isn't the fastest language around. You should be patient when trying to do such huge bulks.

[penguinaka]

thats somewhat true... you can tell the difference between the programs that are trully hung and the ones that are not by the movement/change of the memory in the private working set field.

If it stays the same and doesn't change at all over say a 5-30min time span. You can also go to the performance tab and click the resource monitor button down at the bottom then go to the through the different tabs and it will give you indicators as well if the program is truly frozen or just working things out in memory.

[speakingtohe]

I have an I7 with12 gig Ram and one book usually converts instantly. 10 books can take minutes. 100 books seem to take forever and calibre stops responding several times during this process. Just curious how long 3,000 books would take.
Helen

[theducks]

Originally Posted by speakingtohe

I have an I7 with12 gig Ram and one book usually converts instantly. 10 books can take minutes. 100 books seem to take forever and calibre stops responding several times during this process. Just curious how long 3,000 books would take.
Helen

You are probably just Disc I/O bound
Pant Pant...'Could you slow down on the conversions for Petes sake '
I7. Gimme a break

[speakingtohe]

You are probably just Disc I/O bound

Gotta get that SSD

[theducks]

Originally Posted by speakingtohe

Gotta get that SSD

Your CPU I/O channel is still probably 66MHz, even if you have the latest over-clocked 'I 7'

A Striped RAID does help with Rotational Latency caused Write Issues

[speakingtohe]

Your CPU I/O channel is still probably 66MHz, even if you have the latest over-clocked 'I 7'

A Striped RAID does help with Rotational Latency caused Write Issues

Aaaaggh! I thought it was 133 Mhz. Oh well what do I know.
Going to work in the Yukon for a few months with only a couple of antique laptops, so desktop will look better in the fall
Helen

[theducks]

Originally Posted by speakingtohe

Aaaaggh! I thought it was 133 Mhz. Oh well what do I know.
Going to work in the Yukon for a few months with only a couple of antique laptops, so desktop will look better in the fall
Helen

There is more than one 'Bus' in a PC. PCI and ISA are slow
And I could be out of date , my newest PC has a P4
(No PCI-e, so I don't know PCI-E speeds)