Sounds like a bug in Xvfb in that case. Unfortunately, I dont think X is very actively maintained anymore, so it probably wont be fixed anytime soon.
As I said, it worked for me with tigervnc, so try that. From what I know of VNC servers (admittedly not very much), tigervnc is the most actively maintained one.
https://wiki.archlinux.org/index.php/TigerVNC