I have always had trouble with ć and č in rtf, so I usually convert any Croatian text into a PDF, and jack up the font and cut off the margins to accommodate the smaller screen size (I just use word or open office to do that, I adjust the fonts and margins and then print to pdf). Sorry, I know that probably am not helping too much! I think that there is some sort of way to get the Croatian letters onto an epub. Try hitting the "transliterate unicode characters to USCI" button in Calibre (under look and feel when you use the converter in calibre) and see what happens. (disclaimer: I haven't yet tried that, so take my advice with a grain of salt
).