I do that sometimes but most of the time you get one huge xhtml-file which you may want to split in separate files which takes time, and Calibre-conversions seem to work almost always.
But I just tried doing it with this book, and the font-size in the resulting epub retained the same font size (0.83rem).
But why did Calibre change it? Still curious...
Thanks.