On my Mac, the error message comes back as soon as it's dismissed, I can't even change the port back, or exit. I have to force quit to get out of it.
I use port 8888 for Calibre and port forward port 80 traffic from my router. Log into your router's configuration interface (commonly done via
http://192.168.x.1 - where x is a 0 or 1 - consult your manual if you're not sure) and look for a "virtual server" or "port forwarding" setting. The exact wording is different for different brands, but you want to send inbound or external port 80 or http traffic to port 8080 (or whatever Calibre's using) on the internal IP address of the machine that's running Calibre. If you're not using a static IP address for that computer, you'll need to either do that or set up a DHCP reservation (assuming your router is also a DHCP server).
It's also convenient to use a dynamic DNS service (many routers can do this, or you install an updater program on your computer), so that you don't have to know your external IP address to access Calibre.