I've encountered the same problem, and haven't found a fully satisfactory solution. I believe the main problem is readers/browsers not interpreting relative heights with respect to the page height, otherwise this should (if I'm not mistaken) work fine:
Code:
div.illustration {
text-align: center;
}
div.illustration img {
max-height: 100%;
max-width: 100%;
}
<div class="illustration">
<img src="file.jpg" />
</div>
But all I see is browsers/readers just ignoring the max-height (probably because they assume the <div> can be resized as large as needed, and don't consider the page restrictions), and Prince XML seems to calculate the max-height relative to the page width (so the image will be, at most, as high as the page is wide).
... or maybe I'm not interpreting the max-height correctly?