Unfortunatly no. Rar is not free Software. They allow extraction but compression is only possible with Rar itself. I don't own it and even then, it would have to be called from outside calibre and would only work for people with rar.
Don't know why you have problems with zip. I don't own a mac, but zip is much more common than rar and should be working on every system. Maybe there is some error in the original files you use. Rezipping them could maybe help in the same way as compressing them to rar. Or you could zip them uncompressed. You don't save much space by compressing jpgs, because they are already compressed in a special way for pictures.
As I said, I don't know Mac OS, but the last possibility would be, to write a small batch script (action script on mac?) which you run via "open with" on the zips. Calibre has command line tools, with which you could add the cbr to the library. On windows it would look something like this:
Code:
md "%~dp1convert"
%~d1
cd "%~dp1"
7z.exe x "%~n1.cbz" "\convert"
cd convert
rar whatever
set dir=%~p1
for /F "tokens=2 delims=(" %%i in ("%dir%") do set id=%%i
set id=%id:~0,-2%
calibredb add_format %id% "%~n1.cbr"
cd..
rmdir /s /q convert