Thanks all. The speed increase was really something rather simple (large speed increases usually are). It was a piece of poorly designed code that I wrote back in the day when I had no idea calibre (and the libraries it manages) would become so large.
And I have to say that credit for the speed increase should go at least partially to a user (elfchief) who opened a ticket and did a bit of analysis on his large library to pinpoint the cause of the slowdown. See
See
http://bugs.calibre-ebook.com/ticket/5158