You can load the rtf's onto the reader without conversion. For the htmls you can use Calibre or BookDesigner.
I don't see what the problem is using Calibre to convert. The only problem with Calibre is if you load some books using Calibre and some using Sony Library. Personally, I have done that many times without problem, but some have reported that it is not that good of an idea.
The load time problem you report with Calibre is to be expected. The Sony Library has an algorithim that can paginate books during the transfer. Calibre does not do that so the books are paginated on the reader the first time you open them.