Thanks for all suggestions, all running fast and smoothly now! yay
I deleted the"VLs" composite, then restarted... tested (it wasn't the culprit). Then deleted some other composites which I use mostly as aliases for other columns... again restarted and tested... nothing. Did this process three times I think.
Now when I add the tags, even in bulk, they get added quite instantly, and Calibre too refreshes in less than a sec.
Now to find the culprit, I'm doing things very carefully.