MobileRead Forums - View Single Post

Down South · 06-07-2013, 02:59 PM

I ran some performance comparisions to try and shed some light on
what was happening and possible solutions.
I used 6 drive configurations ranging in speed from an OCZ Diesel 8GB
USB flash drive to 2 Samsung 830 - 128GB SSD's in RAID 0.
I tested with and without a 1.5GB RAM drive for TEMP files and the metadata.db.

Using Process Monitor to see what Calibre was doing and Process Explorer
to see performance and contol CPU affinity allowed me to look at what was
happening when opening the library. (Sysinternals.com)

PROCESSING the data in the metadata.db file which is read in 1K chunks
is what takes all the time.
Any reasonably fast drive will give equal times to open or refresh Calibre.
Let me rephrase that. Use any drive you like. The speed will be the same.

Copying JUST the 51 MB metadata.db file from my 38,000 record library
to 6 different devices and then opening or refreshing the library
on each device took ~17 seconds (same as opening the actual library).

Underclocking and overclocking slow and speed up Calibre by
proportionate amounts of time.
CPU 50% faster 2.4GHZ -> 3.6 GHZ (3/2) = opens in 2/3 the time. ~11 secs.
CPU 2/3 the speed 1.6 GHZ = 3/2 the opening time. ~25 secs.

Calibre/SimpleSQL? uses only 1 processor/core so an Intel Q6600 (My CPU)
(4X 2.4 GHZ) is SLOWER than a single core @ 3 GHZ of the SAME family.
This can be seen in Process Explorer and CPU affinity from 1-4 CPU's
can be assigned to Calibre with benchmarks remaining the same.

Perhaps the more robust version of SQL uses Multiple CPU's?
If so I will be happy to contribute so Kovid can buy and use a
database engine that can multiprocess and open my library 3-4 times faster.

Please do NOT take my word for it. Run your own benchmarks and report back.
I live on a mountain in Mexico and only get to town for the internet every 7-14 days so please do not expect a rapid response from me.

Kovid: Thank you for providing such a fine, multifaceted program.
I KNOW you never expected it to take so much of your time and to
grow to what is has become which is a program being used far beyond it's
original design parameters.

If I remember correctly full versions of SQL ran on 8 or more CPU's and handled 100's of thousands of records with a search or update time under 5 seconds.
That is what we had in the mainframe world at clients like State Farm and AT&T.
I doubt the version of SQL in Calibre is designed to do so.

06-07-2013, 02:59 PM	#1
Down South Junior Member Posts: 7 Karma: 10 Join Date: Jun 2013 Device: Kindle Keyboard	Large Library Performance Comparisons I ran some performance comparisions to try and shed some light on what was happening and possible solutions. I used 6 drive configurations ranging in speed from an OCZ Diesel 8GB USB flash drive to 2 Samsung 830 - 128GB SSD's in RAID 0. I tested with and without a 1.5GB RAM drive for TEMP files and the metadata.db. Using Process Monitor to see what Calibre was doing and Process Explorer to see performance and contol CPU affinity allowed me to look at what was happening when opening the library. (Sysinternals.com) PROCESSING the data in the metadata.db file which is read in 1K chunks is what takes all the time. Any reasonably fast drive will give equal times to open or refresh Calibre. Let me rephrase that. Use any drive you like. The speed will be the same. Copying JUST the 51 MB metadata.db file from my 38,000 record library to 6 different devices and then opening or refreshing the library on each device took ~17 seconds (same as opening the actual library). Underclocking and overclocking slow and speed up Calibre by proportionate amounts of time. CPU 50% faster 2.4GHZ -> 3.6 GHZ (3/2) = opens in 2/3 the time. ~11 secs. CPU 2/3 the speed 1.6 GHZ = 3/2 the opening time. ~25 secs. Calibre/SimpleSQL? uses only 1 processor/core so an Intel Q6600 (My CPU) (4X 2.4 GHZ) is SLOWER than a single core @ 3 GHZ of the SAME family. This can be seen in Process Explorer and CPU affinity from 1-4 CPU's can be assigned to Calibre with benchmarks remaining the same. Perhaps the more robust version of SQL uses Multiple CPU's? If so I will be happy to contribute so Kovid can buy and use a database engine that can multiprocess and open my library 3-4 times faster. Please do NOT take my word for it. Run your own benchmarks and report back. I live on a mountain in Mexico and only get to town for the internet every 7-14 days so please do not expect a rapid response from me. Kovid: Thank you for providing such a fine, multifaceted program. I KNOW you never expected it to take so much of your time and to grow to what is has become which is a program being used far beyond it's original design parameters. If I remember correctly full versions of SQL ran on 8 or more CPU's and handled 100's of thousands of records with a search or update time under 5 seconds. That is what we had in the mainframe world at clients like State Farm and AT&T. I doubt the version of SQL in Calibre is designed to do so.