Is it important that the heading actually be a <hx> tag? (e.g. you want it to appear as a sub-header in a nested toc.ncx) or could you substitute a <p> tag with the same styling as you were using for the <h5> tag? (and you could still put the entry into the toc.ncx by hand.)
And yes, the above is an impure thought, semantics-wise (because I'm suggesting you code for how you want the element to appear, rather than what the element is), but hey, when you are dealing with iBooks you gotta do what you gotta do.
