I'm getting a somewhat similar issue to you when I switch to Segoe UI 11pt: My boolean "Read" column will shrink to the size of the column header, but no narrower.
My other columns (e.g. various text, integer, and datetime columns) don't have this issue.
I also don't have this issue at all on my font of choice (Calibri 11pt).
Maybe try a different font; as I recall, Segoe UI has some sort of weird optimized rendering in Windows that might be mucking things up. I like the
MS ClearType fonts myself. Yes, I deliberately picked Calibri for its name.