I don't want to cause any alarm, but if you've not already looked into this you might want to check your
hard drive health.
When my hard drive started to fail (damn Seagates), the Calibre library on it started to chug badly like the way you describe. I moved it to a good drive and it was back to a nice speed.