Your selected output profile in calibre has a lot to do with the base (default) font size of converted documents.
Also note that calibre has a font scaling feature that you can use for conversions. There's a wizard in the Look & Feel section of the conversion dialogue that allows you to "map" sizes from the input document to the output document.
Calibre user manual (font rescaling)