Apparently Mobipocket didn't distinguish between Latin Serbian and Cyrillic Serbian and "sr" defaulted to "sr-cyrl". I.e. Cyrillic letters.
What you can try to do is open your dictionary with the Java Mobi Metadata Editor
and double-check the actual dictionary input language and change it to "sr-latn" or "sr-cyrl".
BTW, even if your dictionary shows up as a Croatian dictionary it should still work as long as your Serbian books have the same language code as the dictionary input language code. (The main dictionary document language is irrelevant.)