Hmm that looks like Qt is having trouble calculating text metrics. Not much I can do about that in calibre code, but Qt is supposed to be working on improving HiDPI support. You can try the calibre builds here:
https://download.calibre-ebook.com/betas/
They are built with a newer version of Qt (warning those are beta builds -- they are also built with a newer version of Visual Studio, so I'd suggest trying the portable version and seeing if it makes a difference first.)