MobileRead Forums - View Single Post

audeojude · 03-18-2013, 10:50 PM

I thought it would be interesting to report on calibre performance. I just purchased and built a new modern computer with a lot of resources and right now have a good contrast with my old system that was a top end system from about 6 or 7 years ago.

Library size is about 120,000 books, 250 gigs, and a DB file 166 megabytes in size. The directory structure in the calibre library folder has tens of thousands of directories in it which was a huge performance hit.

Operating system is Ubuntu Linux 12.04 for both computers. Just moved the same OS drive from one computer to the next. OS drive is a 100 gig ocz 4x PCI-E ssd raid card with 4 25 gig drives on it in a stripped linux software raid configuration. 200 megs a second through put on the old computer 600 megs a second throughput with the new system.

Last system was a Intel Core2 Quad Core system running at 2.66 ghz with 8 gigs of ram with calibre living on a 3 terrabyte sata 3 internal drive that was on a intell sata 2 hard drive controller.

Opening up calibre took about 2 minutes. A search would take anywhere from a minute to a minute and a half+ This was faster than when I had the library on a older sata 2 hard drive.

New system is a Intel i7-3770T 45 watt cpu Running at 2.5 GHZ with 32 gigs of ram on a z77 motherboard with sata 3 controllers. The 3 terabyte drives have been moved over into that system.

Opening up calibre now takes about 30 seconds to 45 seconds and a search is anywhere from 15 to 40 seconds. After upgrading the system to 32 gigs of ram calibre is using just over 6 gigs of ram to run. CPU usage seems to max out at about 10 to 12 % averaged over all 8 hyper threaded cores when doing a database check or other intensive search. Several cores hitting 100% for extended periods but not slowing the overall system.

My system typically consumes about 15 watts of power idling at the desktop just doing light internet usage. Running a search or database check will pushed the cpu cores to about 10 to 12% and doubled power consumption to about 30 watts. Old system consumed about 130 watts idling.

I just found it interesting the difference in performance in calibre between the systems. I think that most of the performance gain is simply IO going from sata 2 to sata 3. I have future plans to get a large 500 gig ssd specd to around 500 megs a second. just for calibre. that should speed up access 3 or 4 fold if I'm right.

I'm way at the extreme end for number of files and size of database and calibre just chunks through it. It definitely impacts performance once the library grows past about 20000 books but even up to the 120000 I have it is reliable if a bit slow. Again I think a lot of the slowness is due to simple IO with a huge directory structure. Neither Linux nor windows like large numbers of directory's and files in a root directory. I have looked a multiple file system formats to see if I could use something other than NTFS or EXT4 but they seem to be a bit more robust and get decent average benchmarks even against some of the exotic file systems out there.

Just my thoughts on the comparison between the systems.

03-18-2013, 10:50 PM	#1
audeojude Connoisseur Posts: 66 Karma: 14170 Join Date: Oct 2011 Device: kindle 1	Calibre performance I thought it would be interesting to report on calibre performance. I just purchased and built a new modern computer with a lot of resources and right now have a good contrast with my old system that was a top end system from about 6 or 7 years ago. Library size is about 120,000 books, 250 gigs, and a DB file 166 megabytes in size. The directory structure in the calibre library folder has tens of thousands of directories in it which was a huge performance hit. Operating system is Ubuntu Linux 12.04 for both computers. Just moved the same OS drive from one computer to the next. OS drive is a 100 gig ocz 4x PCI-E ssd raid card with 4 25 gig drives on it in a stripped linux software raid configuration. 200 megs a second through put on the old computer 600 megs a second throughput with the new system. Last system was a Intel Core2 Quad Core system running at 2.66 ghz with 8 gigs of ram with calibre living on a 3 terrabyte sata 3 internal drive that was on a intell sata 2 hard drive controller. Opening up calibre took about 2 minutes. A search would take anywhere from a minute to a minute and a half+ This was faster than when I had the library on a older sata 2 hard drive. New system is a Intel i7-3770T 45 watt cpu Running at 2.5 GHZ with 32 gigs of ram on a z77 motherboard with sata 3 controllers. The 3 terabyte drives have been moved over into that system. Opening up calibre now takes about 30 seconds to 45 seconds and a search is anywhere from 15 to 40 seconds. After upgrading the system to 32 gigs of ram calibre is using just over 6 gigs of ram to run. CPU usage seems to max out at about 10 to 12 % averaged over all 8 hyper threaded cores when doing a database check or other intensive search. Several cores hitting 100% for extended periods but not slowing the overall system. My system typically consumes about 15 watts of power idling at the desktop just doing light internet usage. Running a search or database check will pushed the cpu cores to about 10 to 12% and doubled power consumption to about 30 watts. Old system consumed about 130 watts idling. I just found it interesting the difference in performance in calibre between the systems. I think that most of the performance gain is simply IO going from sata 2 to sata 3. I have future plans to get a large 500 gig ssd specd to around 500 megs a second. just for calibre. that should speed up access 3 or 4 fold if I'm right. I'm way at the extreme end for number of files and size of database and calibre just chunks through it. It definitely impacts performance once the library grows past about 20000 books but even up to the 120000 I have it is reliable if a bit slow. Again I think a lot of the slowness is due to simple IO with a huge directory structure. Neither Linux nor windows like large numbers of directory's and files in a root directory. I have looked a multiple file system formats to see if I could use something other than NTFS or EXT4 but they seem to be a bit more robust and get decent average benchmarks even against some of the exotic file systems out there. Just my thoughts on the comparison between the systems.