I upgraded to 6 on my iPhone 4 when it was released. The actual upgrade experience was bad - the wifi download crapped out several times, and when I upgraded via iTunes, it told me the download was corrupted, and my phone was stuck in restore mode for about two hours. I was convinced it had bricked, but on the fourth go-round it worked.
In terms of the iOS6 experience...meh. Obviously, with the iPhone 4, you still don't have Siri, and you don't get the Panorama feature on the camera. Google maps disappears, and is replaced with Apple's Maps, which is (IMHO) an inferior product (yes, you can still use Google Maps via a web shortcut). I, personally have zero use for Passbook. Also, for the first week after upgrading, my battery life was terrible. With my usual phone usage, my battery would be at about 15-20% when I went to bed at night (I leave wifi and GPS on at all times, bluetooth always off). With no change in usage, my battery would be down to 10% by the time I came home from work at 4pm. I solved this by going into notifications and turning off a bunch of stuff and restricting which apps use location services (I think Passbook might have been a culprit).
So, my iOS6 experience has been mediocre. I really liked the move from iOS4 to iOS5, however. So, to sum it up, knowing what I know now, if I were at iOS5, I would probably stay there, but if I were at iOS4, I would probably go ahead and jump to iOS6.
Obviously, this is merely my own personal experience, and I'm sure others here love iOS6.