Thanks for your reply.
I actually found out what the problem is, while I waited for a reply
Looks to me that the new standalone server is not reading the settings from the server settings inside calibre, so when I set the port to be 8081, the standalone server still listen on 8080.
So all I need to know now is how to make the standalone content server listen on port 8081?
As I said, I use nssm to make the standalone server start as a service, so if you have any tips for me it would be highly appreciated.