I used
codecogs to generate the formulas in SVG. If specify the height and width in em, they scale when you zoom in the text. All ePUB readers should support SVG.
If you want to use images, the quality is often less. If you want to use images, use png. For text-based images that will result in better quality.