I would strongly recommend looking for an alternative to Calibre for ebook creation. Although I strongly doubt you can get much smaller files, unless you're doing something really weird.
In my opinion / limited experience, Calibre was created for the purpose of converting & syncing ebooks with a personal e-reader. I would look at special purpose software like Sigil instead.
Search for "epubcheck" and "epub preflight" if you haven't seen them already.
I dunno much about PDF. You should make sure you know exactly what you want of it. Here are some possible questions to think about - do you want people to be able to print it? Or does it need to be readable on a 5" e-reader without reflowing? Do you want reflowing to work?