It is hard to help you without examining the document, but I do not think OpenOffice is the culprit.
Did you have a look on their rtf Style Guide (http://www.smashwords.com/books/down...le-guide.rtf)?
On page 8 there is a "Five Common Formatting Mistakes to Avoid:..." section: you can check to see if it can solve your problem.
If it does not help, I would rip all formatting, just keeping the text (italic and bold also of course) and redo its format following the guidelines provided in their Style Guide. Of course if your text is very long and heavily formatted this will be time consuming, but if it's what it takes!
Anyway, good luck!