The last time I was looking for a multi-format reader for PC, I settled on
FBReader. It's light on system resources (and therefor fast) and highly configurable. For PDFs you're probably going to want a dedicated PDF reader--I use Sumatra PDF, again mainly because it's lightweight and fast.
In my experience, when you try to use Calibre "only" for a simple task, whether reading or one-off conversion or loading some books on a mobile device, you're in for headaches. When you're ready to really dig in and organize a large library and straighten out metadata and keep devices synced (and/or set up content fetching and serving, which I haven't tried), you'll appreciate that Calibre is as powerful as it is, but until you get to that point it can feel like learning how to operate a front-end loader in order to drive in a nail.