optimisign conversion speed - use 2 hard drives ?

cybmole · 10-12-2010, 03:58 AM

I know that some programs e.g. my Internet Download Manager, can be optimised by using a 2nd hard drive for temporary file storage, rather than having all disc I/O take place on the same HD.

is this true for calibre ? - I know that it writes lots of fiels during conversion, so is there any benefit ( & any prefs setting ) which would place these on a different drive than the one which holds the calibre library ?

I also don't really get how the default of 6 threads should be optimised. is the best number a function of PC cores, or of IO speed or what ?

Manichean · 10-12-2010, 04:39 AM

This assumes that for conversion (or your download manager), the bottleneck is hard drive speed. I don't think that's the case in either program, so I'd be really surprised if it worked on your download manager. As for Calibre, I know it makes no discernible difference, as I have that exact setup (by coincidence) on one of my computers.

Edit to add: If the issue was the hard drive speed, you'd be way better off installing the library and temporary files on a RAID than using a second hard disk.

itimpi · 10-12-2010, 05:00 AM

Quote:

Originally Posted by Manichean

Edit to add: If the issue was the hard drive speed, you'd be way better off installing the library and temporary files on a RAID than using a second hard disk.

Not normally true if using something like RAID5 as it has slower write speeds than an individual disk as writes to all drives in the array need to complete before the write succeeds.

Manichean · 10-12-2010, 05:20 AM

Quote:

Originally Posted by itimpi

Not normally true if using something like RAID5 as it has slower write speeds than an individual disk as writes to all drives in the array need to complete before the write succeeds.

Well, I wasn't talking about any specific RAID, as you'll notice. I think a RAID0 would be preferrable here.

itimpi · 10-12-2010, 05:28 AM

Quote:

Originally Posted by Manichean

Well, I wasn't talking about any specific RAID, as you'll notice. I think a RAID0 would be preferrable here.

Fair enough - but most people seem to go for RAID for resilience reasons rather than performance (and those RAID variants tend to be slightly slower) which was why I made the comment. In fact I suspect many people do not realise the trade-offs between performance and resilience offered by the different types of RAID.

Manichean · 10-12-2010, 05:33 AM

Quote:

Originally Posted by itimpi

Fair enough - but most people seem to go for RAID for resilience reasons rather than performance (and those RAID variants tend to be slightly slower) which was why I made the comment. In fact I suspect many people do not realise the trade-offs between performance and resilience offered by the different types of RAID.

That may be, but we are talking about optimizing speed here

cybmole · 10-12-2010, 06:34 AM

Quote:

Originally Posted by Manichean

This assumes that for conversion (or your download manager), the bottleneck is hard drive speed. I don't think that's the case in either program, .

Well the IDM devlopers themselves recommend it!

they say this
" Temporary directory is required for storing file parts during download. If you have different physical drive on your computer you should select different physical drives for temporary directory and for" save to" folders for faster assembling of downloaded files "

( A download manager opens multiple simultaneous connections & thus ends up with many small file fragments which then have to be assembled into a single file once the download is complete. )

& as I understand it a calibre conversion involves creating lots of small temporary files, then using them to build the final output ?

changing the PC into a RAID setup is mega bucks overkill just to shave a few secs off of an occasional bulk conversion, surely ?

Perkin · 10-12-2010, 07:01 AM

I think the problem is the size of the files, with the download manager, you're probably talking about multi megs/ gig files, so assembling a file from hundreds of 'pieces' into a complete large file will be quicker when copying to an alternate physical drive, than it would assembling it to the same drive as the pieces are on.

With the calibre files, usually the size is maximun 4-5 Meg, and this wouldn't make much improvement speedwise using a seperate drive for temp files, as the bottleneck is the processing of the file, not the copying.

Manichean · 10-12-2010, 07:05 AM

Quote:

Originally Posted by cybmole

Well the IDM devlopers themselves recommend it!

they say this
" Temporary directory is required for storing file parts during download. If you have different physical drive on your computer you should select different physical drives for temporary directory and for" save to" folders for faster assembling of downloaded files "

( A download manager opens multiple simultaneous connections & thus ends up with many small file fragments which then have to be assembled into a single file once the download is complete. )

& as I understand it a calibre conversion involves creating lots of small temporary files, then using them to build the final output ?

changing the PC into a RAID setup is mega bucks overkill just to shave a few secs off of an occasional bulk conversion, surely ?

You're the one who started this discussion...

As for the download manager, assembly of files would likely be faster on different drives, but that's only a tiny part of the overall operation, that is downloading the file in the first place. While downloading, the bottleneck is surely the internet connection. You do the math.

And as far as Calibre's conversion process goes, yes, it creates temporary HTML files. I'm no expert on that, but I'd guess the main cost here is processing the files, not the deassembly of the source format and reassembly of the target format. So, I'd guess that the conversion speed would profit more from having a faster CPU and access to more RAM as opposed to the temporary files being on a different disc. As I mentioned before, I have that exact setup by coincidende on one of my computers, and it doesn't make any discernible difference in conversion times.

And, really, speeding up conversion? Most of my conversions took well under a minute.

cybmole · 10-12-2010, 07:18 AM

ok thanks - let's drop the separate drives issue -

what about the number of threads setting. why is the default value 6 & does tweaking that make any difference? ( to when calibre is given say 10+jobs to be getting on with )
FYI I have a core duo PC with 2 x 2.7 ghz cores & 2GB RAM

Manichean · 10-12-2010, 07:31 AM

Quote:

Originally Posted by cybmole

ok thanks - let's drop the separate drives issue -

what about the number of threads setting. why is the default value 6 & does tweaking that make any difference? ( to when calibre is given say 10+jobs to be getting on with )
FYI I have a core duo PC with 2 x 2.7 ghz cores & 2GB RAM

As far as I know, Calibre only has at most as many worker threads active as there are CPU cores. The other threads idle away, I think.
For reference purposes: I have just started Calibre, and I have the default settings concerning threads. My process manager shows one instance of calibre.exe and two instances of calibre-parallel.exe, I haven't interacted with the program at all, and the server isn't running either.

cybmole · 10-12-2010, 07:42 AM

the "6" value is at preferences advances misc. it's now called max number of waiting worker processes. below that is a tick box option to match processes to cores.

purely out of academic curiosity, I was wanting to understand those options better.

Manichean · 10-12-2010, 08:01 AM

That says it all, doesn't it? It's the maximum number of processes waiting to be able to do stuff. The maximum number of working processes thus would be equal to the number of CPU cores.

chaley · 10-12-2010, 08:16 AM

Quote:

Originally Posted by cybmole

the "6" value is at preferences advances misc. it's now called max number of waiting worker processes. below that is a tick box option to match processes to cores.

My feeling (I haven't verified any of this so I can't call it an understanding) is that if you are doing CPU-intensive things like conversions, you want to check that box. Having extra processes waiting for the CPU simply slows things down even more. Thinking of how O/S schedulers I have built behave, I would consider setting the number to 1 more than the number of cores so that I/O time can be soaked up by a waiting process.

If you are doing network-intensive things like news and metadata downloads, then you do not want to check the box. Each process can get as much CPU as it wants. Of course, it can't get as much network bandwidth as it wants. My experience has been that for most people, the bandwidth issues are not on the local end, so it makes sense to have multiple clients. (Multi-channel downloaders depend on this phenomenon.)

My guess is that adding books falls into yet another category. It would be I/O-bound, not CPU bound. However, running lots of I/O-bound processes has its own drawbacks, including failed seek optimizations and thrashing though disk caches. I would probably use a number lower than my number of cores.

kovidgoyal · 10-12-2010, 10:52 AM

just to shed some light on the the number of processes issue: calibre maintains two separate worker process pools. One is used for adding books/saving to disk and the other for conversions. So if you set it to 6 that means a maximum of 3 conversions will run simultaneously. And that is why you will see the number of worker processes changes by two when you use the up and down arrows.

In addition to this some conversion plugins run tasks in their own poll of processes, so for example if you bulk convert comics, each comic conversion will use three separate processes to render the images. The job manager knows this so it will run only a single comic conversion simultaneously.

And since I'm sure someone will ask: The reason adding/savig books are in separate processes is because of PDF. PDF processing libraries can crash on reading PDFs and I dont want the crash to take down all of calibre. Also when adding EPUB books, in order to extract the cover you have to sometimes render the HTML of the first page, which means that it either has to run the GUI thread of the main proces or in a separate process.

Finally, the reason calibre keep workers alive and idle instead of launching on demand is to workaround the slow startup time of python processes.

10-12-2010, 03:58 AM	#1
cybmole Wizard Posts: 3,720 Karma: 1759970 Join Date: Sep 2010 Device: none	optimisign conversion speed - use 2 hard drives ? I know that some programs e.g. my Internet Download Manager, can be optimised by using a 2nd hard drive for temporary file storage, rather than having all disc I/O take place on the same HD. is this true for calibre ? - I know that it writes lots of fiels during conversion, so is there any benefit ( & any prefs setting ) which would place these on a different drive than the one which holds the calibre library ? I also don't really get how the default of 6 threads should be optimised. is the best number a function of PC cores, or of IO speed or what ?

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Calibre does not see external usb drives.	WEMV	Calibre	6	02-07-2010 04:20 PM
PDF drives me crazy!	Ektoplasma	Workshop	3	06-17-2008 09:04 PM
Easy hard drive data archiving with a USB hard drive adapter	Bob Russell	Lounge	24	02-20-2007 04:15 PM
"Perpendicular recording" technology advances! Hard drives still yet to hit market	Alexander Turcic	Lounge	3	06-11-2005 03:15 AM

10-12-2010, 04:39 AM	#2
Manichean Wizard Posts: 3,130 Karma: 91256 Join Date: Feb 2008 Location: Germany Device: Cybook Gen3	This assumes that for conversion (or your download manager), the bottleneck is hard drive speed. I don't think that's the case in either program, so I'd be really surprised if it worked on your download manager. As for Calibre, I know it makes no discernible difference, as I have that exact setup (by coincidence) on one of my computers. Edit to add: If the issue was the hard drive speed, you'd be way better off installing the library and temporary files on a RAID than using a second hard disk.

10-12-2010, 07:01 AM	#8
Perkin Guru Posts: 655 Karma: 64171 Join Date: Sep 2010 Location: Kent, England, Sol 3, ZZ9 plural Z Alpha Device: Sony PRS-300, Kobo Aura HD, iPad (Marvin)	I think the problem is the size of the files, with the download manager, you're probably talking about multi megs/ gig files, so assembling a file from hundreds of 'pieces' into a complete large file will be quicker when copying to an alternate physical drive, than it would assembling it to the same drive as the pieces are on. With the calibre files, usually the size is maximun 4-5 Meg, and this wouldn't make much improvement speedwise using a seperate drive for temp files, as the bottleneck is the processing of the file, not the copying.

10-12-2010, 07:18 AM	#10
cybmole Wizard Posts: 3,720 Karma: 1759970 Join Date: Sep 2010 Device: none	ok thanks - let's drop the separate drives issue - what about the number of threads setting. why is the default value 6 & does tweaking that make any difference? ( to when calibre is given say 10+jobs to be getting on with ) FYI I have a core duo PC with 2 x 2.7 ghz cores & 2GB RAM

10-12-2010, 07:42 AM	#12
cybmole Wizard Posts: 3,720 Karma: 1759970 Join Date: Sep 2010 Device: none	the "6" value is at preferences advances misc. it's now called max number of waiting worker processes. below that is a tick box option to match processes to cores. purely out of academic curiosity, I was wanting to understand those options better.

10-12-2010, 08:01 AM	#13
Manichean Wizard Posts: 3,130 Karma: 91256 Join Date: Feb 2008 Location: Germany Device: Cybook Gen3	That says it all, doesn't it? It's the maximum number of processes waiting to be able to do stuff. The maximum number of working processes thus would be equal to the number of CPU cores.

10-12-2010, 10:52 AM	#15
kovidgoyal creator of calibre Posts: 43,843 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	just to shed some light on the the number of processes issue: calibre maintains two separate worker process pools. One is used for adding books/saving to disk and the other for conversions. So if you set it to 6 that means a maximum of 3 conversions will run simultaneously. And that is why you will see the number of worker processes changes by two when you use the up and down arrows. In addition to this some conversion plugins run tasks in their own poll of processes, so for example if you bulk convert comics, each comic conversion will use three separate processes to render the images. The job manager knows this so it will run only a single comic conversion simultaneously. And since I'm sure someone will ask: The reason adding/savig books are in separate processes is because of PDF. PDF processing libraries can crash on reading PDFs and I dont want the crash to take down all of calibre. Also when adding EPUB books, in order to extract the cover you have to sometimes render the HTML of the first page, which means that it either has to run the GUI thread of the main proces or in a separate process. Finally, the reason calibre keep workers alive and idle instead of launching on demand is to workaround the slow startup time of python processes.

Advert

Advert