I don't see why with that configuration you should be seeing the times you do, unless you have zillions of tags and authors. That said, the next release has changes that will help. First, closing the tag pane will stop all the work needed to update it. Second, the time to update the tags pane has been reduced by a factor between 5 and 6 times.
We will see whether these helps you, or whether your machine is haunted.
