For what file system has the drive been formatted. And can you confirm that calibre start up really takes 51 minutes
Yes its worth moving the libraries to another drive to see what happens. Have you run chkdsk on that 2TB drive - nothing you have posted has removed my suspicion that its a hardware issue.
If you install Process Explorer (get it from MS) and have a look at the hardware interrupts entry then if its much more than 1% CPU usage then it probably indicates something is failing, see attachment. also download the disk manufacturers diagnostics tools and run those.
BTW if its an NTFS formatted drive you don't need to defrag, Windows 7 does it on the fly - if you use a 3rd party defragger then I suggest you turn off the Windows on the fly defragging - Google will find you plenty of help with that.
Is it possible that you are running multiple AV's - I recently stumbled on a system that was running MSE, McAfee and AVG - calibre was sluggish but so was everything else - 10 minutes to boot and 20 minutes to shut down.
I run calibre on a lesser system than you are using - 3.2GHz i5, 6GB Ram and a 2TB internal Sata 2 drive, I allow MSE AV to scan the calibre libraries, the entire drive is indexed, and the drive is compressed - I have no performance issues.
are you running an image manager that sucks up any images it can find - such as Picasa or similar? You may not be running the main program but they can hide their scanners - Process Explorer or Task Manager should help you spot anything like that.
Task Manager also has some useful graphs for monitoring resource usage - ctrl/shift/esc - click Performance Tab, click Resource Usage button.
BR