Hello Kovid,
x11vnc is just the gateway to a running X server, or starts Xvfb if there is no running server. I haven't used vino, but it looks the same as x11vnc; not an X server but just a VNC<->X11 gateway.
RealVNC (vnc4server) and tightvncserver have their own X server that runs in memory instead of on a physical monitor and directly talks to vnc clients. I can't find tiger vnc client or server in the default ubuntu repositories
but it is in the same category.
I was using x11vnc with the Xvfb server, but ran into the same error. Running x11vnc with the standard X server (/usr/bin/X) did work for me, but meant that my session was displayed on the console in addition to the remote display, which I did not want. I'll have to download tigervnc to see if it works better for me. Worst case, I can create a VM on my linux box if I need to, so that I don't have to dedicate a physical display.
Thank you for creating calibre; it's a great program I use almost every day.
Best Regards,
Jeff