The encoding detection function is failing, not sure if that's a bug or related to some bad data in your rtf file. Non-ASCII support in RTF files is relatively new. You could open a bug on the bug-tracker with a reproduction test file attached:
https://bugs.launchpad.net/calibre
For the RTF file there is a workable way to get a good conversion:
- Open the file in MS Word
- Save/Export the file as HTML (Filtered), make sure the encoding is UTF-8
- Import the HTML file to Calibre, edit metadata as desired, then convert
A pdf file wouldn't be giving you the same problem, so if that's failing you should post the error separately.