Do you know if the images in the book are vector graphics, or rasterized image files? If the latter, I would have guessed that calibre or MobiPocket creator could handle them. I'm not sure if any program currently available can handle vector graphics in PDFs well. Probably if there are math symbols in the text anyway, conversion is not going to go well anyway.
My advice would actually be to keep the file in PDF format and try to make it readable by cropping all the whitespace and dividing each page into reader-sized chunks, maybe with landscape in mind. Perhaps a combination of soPDF, PDFLRF and/or BRISS, depending on the exact layout of the book.
Some advice on that here.