You dont need to install either xvfb or imagemagick. Those were needed only for very old versions of calibre. You do need to have the X server libraries installed, however. So installing xvfb is probably a good way to do that
You should also change the service file to run the server as a normal linux user rather than root. See the section on systemd integration at
https://manual.calibre-ebook.com/server.html