It should be very easy to use only the viewer in calibre. Just associate the file formats with the ebook viewer.
You can read more about it here:
http://manual.calibre-ebook.com/viewer.html
However you might benefit from the knowledge that the calibre ebook viewer actually is a epub viewer. But since calibre has the ability to convert between different formats, calibre can convert other formats to epub "on the fly" and let you read the resulting file in the calibre viewer that way.
But you may improve the reading experience by using the tools in calibre to make a better conversion by making adjustments to the process. You can adjust things like layout, line breaks, paragraph spacing, page breaks, indentation, fonts, table of contents, headings, cover and other things.
Pdf is a special case. It is very difficult to convert. I personally have standardised on just epub and pdf. I convert everything to epub, except for pdfs, and leave pdfs as they are. That way I only have to be able to read epub and pdf.
So my main use of calibre is to fix metadata, convert to epub and then send the epub to my reading device. I don't actually use calibre for much else, but I still keep my books there, as a backup. But I only keep the books in calibre as pdf or epub, all other formats are converted and then deleted. I also save the original unconverted formats outside calibre. Good to have if I discover that something went wrong in the conversion.