The HTC Universal was 5" back in 2005. I had a 3.6" phone in 2006, the size of that device was actually larger than today's 5 inchers. 5" phones easily fit into a man's front pockets of even the tightest jeans.
But I actually went back to 4.5" because I liked the new phone so much, would be great if it was 1/2 larger, of course. But now that I have a tablet I don't use the phone so much. And yes, Iguana is right, making calls is just a minor function of today's smart phones, so 3.5" is almost unusable. And why should I do that to my eyes when a larger phone handles so well?