So I finally got it running properly, with multiple libraries. I had to physically install Calibre on the server system, FTP all my libraries to it. Then launch Calibre, go through the process of assigning all the libraries. Then start the server from within Calibre itself, and not command line.
Now however, I have a new problem. I'm using an https site, but getting the error saying it needs to be an https site?
WARNING: This browser does not support Application Cache so offline access to calibre will not work. Some browsers only enable Application Cache over HTTPS connections, so you may have to setup HTTPS for your calibre server.
Yet, my site is:
https://calibre.mobilefiles4free.com/
What did I do wrong here? LOL
PS: Thank you very much kovidgoyal, for all your help, support and for creating such wonderful software.