You can use netstat on windows to find a list of open connections and the external url those ports are connected to. That is one way to find out what is using your port 8080.
As to the dialogue mentioning the wrong port, it looks like you found a bug in calibre. I would report it on their bug tracker. Whether or not it is simply a label problem or actually not using the port you configured, I don't know.