Multi-Core Conversion support?

evandy · 08-29-2011, 08:04 PM

I was wondering if anyone had started taking a look and re-structuring Calibre's conversion process so that it can take advantage of multi-cores. If I am running a conversion with, say, search-and-replace, and heuristic processing enabled, the conversion can take an extremely long time to complete. Obviously, there is a limit to what can be done without adding hard re-integration problems. But I was thinking that on a multi-core PC, you could realize some useful gains by, say, running the input plugin on one core, piping the output straight to search&replace, piping that to heuristics, and then piping the heuristic output to the output plugin. I would think that this kind of pipelining might be relatively easy to implement... and any gains would be useful at this point.

kovidgoyal · 08-29-2011, 08:53 PM

Nope, no one is taking a look at that. If I want to exercise multiple cores, I run multiple conversions.

It should be fairly easy to use multiprocessing to speed up heuristics and S&R but it's not something I care enough about to move up my TODO list. Feel free to submit a patch

evandy · 08-29-2011, 09:15 PM

I am a father of 2, and the oldest is just 2 years old... I'd be happy to do that, but it might take a couple years to find the time. I've got a single conversion that's been going 160 minutes so far tonight, and still chugging, so you can see why it would be useful.

If it isn't done by the point I have some free time, I'll take a look.

ldolse · 08-30-2011, 02:46 AM

Odds are it's the format scene breaks feature - that function needs some further optimization for some books based on user feedback, but I don't have any test cases where it's slow to know exactly what's wrong (works fine when I use it). Submit a bug request and I'll look into it, but otherwise just disable that one option.

Raimond · 09-18-2015, 10:05 AM

It's been several years since this topic was discussed.
By now, quad and octa core systems are common and the conversion process could benefit a lot.

Any chance that this could be bumped to the TODO list?

I took a look at the conversion process but frankly that's too "magic" for me to do the patching myself.

kovidgoyal · 09-18-2015, 10:49 AM

Nope, still not interested, sorry

As I said before, if I feel the need to exercise multiple cores, I simply convert multiple books. Single books that take a very long time to convert are pretty rare, so spending effort parallelizing individual conversions is really not worth it, as far as I am concerned.

08-29-2011, 08:04 PM	#1
evandy Enthusiast Posts: 25 Karma: 12 Join Date: Jun 2009 Device: Kindle DX	Multi-Core Conversion support? I was wondering if anyone had started taking a look and re-structuring Calibre's conversion process so that it can take advantage of multi-cores. If I am running a conversion with, say, search-and-replace, and heuristic processing enabled, the conversion can take an extremely long time to complete. Obviously, there is a limit to what can be done without adding hard re-integration problems. But I was thinking that on a multi-core PC, you could realize some useful gains by, say, running the input plugin on one core, piping the output straight to search&replace, piping that to heuristics, and then piping the heuristic output to the output plugin. I would think that this kind of pipelining might be relatively easy to implement... and any gains would be useful at this point.

08-30-2011, 02:46 AM	#4
ldolse Wizard Posts: 1,337 Karma: 123455 Join Date: Apr 2009 Location: Malaysia Device: PRS-650, iPhone	Odds are it's the format scene breaks feature - that function needs some further optimization for some books based on user feedback, but I don't have any test cases where it's slow to know exactly what's wrong (works fine when I use it). Submit a bug request and I'll look into it, but otherwise just disable that one option. Last edited by ldolse; 08-30-2011 at 02:49 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Multi-level TOC broken in epub->epub conversion	siebert	Conversion	14	03-09-2011 05:38 PM
Multi language support via Calibre	Jim77	Calibre	4	02-12-2010 02:46 PM
Astak's Mentor Ereader multi-languages support?	dkarale	Astak EZReader	8	01-11-2010 03:39 PM
Does any device support multi-level TOC's?	AnemicOak	ePub	14	10-07-2009 01:08 PM
O'Reilly to support multi-format e-books, goes ePub	Alexander Turcic	News	30	06-20-2008 10:58 PM

08-29-2011, 08:53 PM	#2
kovidgoyal creator of calibre Posts: 43,853 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Nope, no one is taking a look at that. If I want to exercise multiple cores, I run multiple conversions. It should be fairly easy to use multiprocessing to speed up heuristics and S&R but it's not something I care enough about to move up my TODO list. Feel free to submit a patch

08-29-2011, 09:15 PM	#3
evandy Enthusiast Posts: 25 Karma: 12 Join Date: Jun 2009 Device: Kindle DX	I am a father of 2, and the oldest is just 2 years old... I'd be happy to do that, but it might take a couple years to find the time. I've got a single conversion that's been going 160 minutes so far tonight, and still chugging, so you can see why it would be useful. If it isn't done by the point I have some free time, I'll take a look.

09-18-2015, 10:05 AM	#5
Raimond Member Posts: 16 Karma: 10 Join Date: Nov 2010 Device: Sony PRS-600, Kindle Paperwhite	It's been several years since this topic was discussed. By now, quad and octa core systems are common and the conversion process could benefit a lot. Any chance that this could be bumped to the TODO list? I took a look at the conversion process but frankly that's too "magic" for me to do the patching myself.

09-18-2015, 10:49 AM	#6
kovidgoyal creator of calibre Posts: 43,853 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Nope, still not interested, sorry As I said before, if I feel the need to exercise multiple cores, I simply convert multiple books. Single books that take a very long time to convert are pretty rare, so spending effort parallelizing individual conversions is really not worth it, as far as I am concerned.

Advert

Advert