I found a work-around for this. It looks like a Qt issue from this StackExchange question:
http://stackoverflow.com/questions/2...-a-window-icon
After resizing /opt/calibre/resources/images/library.png to 256x256 (it was 1024x1024), I'm now able to start calibre under vncserver.
I'm using the RealVNC 4 vncserver on Ubuntu 16.04. I didn't test it with tightvnc, which is the only other option in the Ubuntu repositories.
The Xkeyboard and Xrandr messages can be ignored since they're only optional X extensions, and not in any of the vnc servers that I've used.