Does your pdf have a background image?
I had a couple of pdf books that sound similar to what you're seeing - the input files were about a megabyte but the calibre-converted epubs were all around 40-50MB. They had a grey background image, and calibre wanted to add the same simple background image to the epub thousands of times. These were pdfs that were exported from OpenOffice direct by the author - they weren't scanned books.
An epub file is just a zip file - you could always rename it to "book.zip" and then look inside with Windows Explorer by double clicking on it to see what's taking up the space.