You can have arbitrary mathjax config, as shown here:
http://manual.calibre-ebook.com/typesetting_math.html
As for your issue, this the first time I've come across, it. Open the demo on that page and you will see no problems with font size on windows. It is something specific to your book. As I said, you will need to contact the MathJax developers for help, or workaround it by scaling as I posted earlier.