Use k2pdfopt - it formats the pdf pretty well (although there are still a few images that are dodge)
http://www.willus.com/k2pdfopt/
Then use calibre to convert to MOBI
Hope that works!
EDIT: Other advice would be to separate chapters by printing to pdf as math textbooks are VERY large.
Also number lines go a bit crazy :P
Also separating tables of values and exercises into one file is a good idea for easy access