I just got my Kindle last week and reading mathematic and lecture PDFs from university on it was the main reason for buying it. Although I'm okay with how it works (though there could be more options to pan, zoom and navigate, especially if a drawing is divided across the 3 zoom areas you get in landscape mode), I also tried to convert a few of those PDFs with Calibre to .mobi out of curiosity.
It's not only math formulas but also drawings that get misconverted. My idea is to improve heuristics to optionally detect drawings and render them to images instead and do the same for any formulas. Detecting formulas in LaTeX-generated PDFs could work quite well based on a recognition of the embedded fonts being used (simply by their embedded name?). Drawings could be detected by image recognition (render PDF page or paragraphs from it to image, run detection algorithm on that image to tell drawings/images from text and then decide how to convert it). It would of course be very slow but since you would convert a book only once, that should be okay. I'm sure a detection algorithm for text vs. graphics already exists in many implementations (OCR or OpenCV).
I'm not sure if anyone attempted to implement a detection/heuristic like that already. If I had some more spare time at the moment I would take a look at it; unfortunately I'm currently too busy, so I don't think I may come up with improvements any time soon. If anyone started/starts doing something like that, I'd be glad to know about it (maybe drop me a PM).