Welcome to MobileRead.
There are lots of great people here that have a lot of knowledge and love to share what they know. They have helped me a lot in the little time I have had a SOny Reader. I am sure that they will do the same for you.
I have found that the best format for me with text files on the Sony Reader is RTF (Rich Format Text.) You are correct that the full line followed by a short line on the Reader when using text files is a result of hard CRs in the document. In another thread there is a
Word Macro that will remove the hard CRs within a paragraph, set the font to the correct size (I favor 16 point, many like 14 point), and remove extra spaces. It also allows you to maintain the two spaces after a sentance if that is to your liking. (I am one that likes only one space after a sentance.)
The PDF problem is well documented in many threads throughout MobileRead and others may be able to guide you better than I can as I have not played around with this aspect as I am quite happy with the results of the RTF method and have not yet moved beyond that.