The viewer in calibre is just a epub viewer. It is possible to view other formats because the are converted "on the fly" to epub and then displayed. To make calibre faster you could make these conversions in advance. That way the ePub version of the book can be displayed at once, without any delay. Or use another reader for other formats.
I only use the calibre reader to check books quickly, not to read. To read I use my Xoom and Mantano.
Almost all actions in calibre are on the database, not on the books in the library. So calibre is very fast, even with a lot of books. The whole database can easily be kept in RAM.
One exception to this is custom columns that displays the formats of the books. For that calibre looks at the actual books to see what formats they have, not just the database. So sorting or searching for books with a specific format can be very slow.
Also complex custom columns that depends on other columns may make calibre slow, because the values of the columns have to be evaluated before they can be displayed.
A discussion about problems with the formats column can be found here:
https://www.mobileread.com/forums/sho....php?p=2461568