Most likely it is the formats column that cause the delay.
Avoid sorting on it, and even consider removing it.
If you sort on it, then Calibre will go through all the books and check each and every file to see what formats you have stored in the library. That may take a while...
More about that here:
https://www.mobileread.com/forums/sho...d.php?t=188312