Well, I missed the part where you said you generated it on your own, anyway
I think you need to systematically try a number of things. First of all: is every RTF you converted using *the same Calibre settings* turned into something 'unreadable' (that is, without tweaking)? Fiddling with different settings won't help to dilucidate the question unless that's the case.
If some files convert OK and some don't (always using the same conversion options), you need to keep in mind that
visually identical documents may be very different in their structures -- that applies to pretty much every format and makes conversion a really hairy issue, and it's only downhill from there.
Sure I can help you with ePub internals and formatting if needed, just drop me a line via PM.