VM doesn't actually do much at all itself, it's just changing Calibre's settings.
I would guess that it has to do with one or more of:
- How many books are in the library
- How many columns you have defined to show
- How many columns are you sorting on
- How fast your computer is
- If you are applying Search/VL options
My views switch in something like 50-100ms I would estimate--it's not instant, but it is quick. I have ~5700 books, a baker's dozen of custom columns, generally only look at 8 columns total and only sort on 3 columns.