This is some incompatibility between Qt 5.6 which calibre 3 uses and your X server. Basically, Qt for some reason gets the wrong/missing DPI information for the screen, which in trun causes it to use absurd/negative sizes for windows, and that causes those errors.
Unfortunately, I know of no way to find the root cause of this. If I had to guess, I'd say it is related to the new high DPI support in Qt. You can running calibre like this, it might work:
QT_SCALE_FACTOR=1 QT_DEVICE_PIXEL_RATIO=1 calibre
|