I have a Dell Streak, which is 5". I feel it is the perfect size for a phone by my definition (the biggest screen size that still comfortably fits into my jeans' front pocket). The Note has almost exactly the same dimensions, it is just a tad wide and a little bit shorter.
I don't feel awkward making calls without a headset, and that "one-handed operation" is a red herring. One-handed works fine for calls, and for almost everything else you hold the phone in landscape mode with one hand.
It is fine for reading in portrait (though I much prefer e-ink), good for quick browsing, as a music player, etc. Once you are used to a big screen, it is hard to go back to 5". But for now 5"+ is the perfect device on the go (until they develop devices with roll-up or foldable screens that could easily double the size).
So as a phone that is the best all-in-one on the market right now? Yes. As a tablet substitute? Not for me.