I don't think so. calibre's codebase is heavily intertwined, and Qt5 is used for various tasks even in the server.
So installing a lot of support software is not going away.
On the other hand, space waste aside, the server itself does not require much in the way of RAM/cpu.
...
Check the Calibre ==> Related Tools subforum, there are a few tools for running a lightweight server e.g. COPS (written in plain old php running on your choice of server).