Will look more into it later, but I'll probably want to keep the annotation count column...
Another thing I noticed is that on my portable install, which has a setup closer to default settings (few plugins etc.), a Calibre restart is needed to get the annotation count updated. OTOH, on my main install, just reopening a recently highlighted book triggers the annotation count update.