Thanks so much for all the suggestions, I really appreciate the help.
Action Chains isn't responsible - I removed all my events/modules/chains and even uninstalled it to test, but still getting the lag when switching VM views manually.
I don't have "switch to VL" enabled in VM... I'm also not sorting by anything (in the VM view settings) except Date or Title.
At this point I think it may be related the number of columns I'm viewing, because when I add more, it does get slower; and faster with fewer columns. But it's still laggy (2-3 seconds) even with only 2-3 basic columns showing (title, author, series).
This might just be something I have to get used to.