I'd think it probably wouldn't affect display life, actually, or they'd have the rascal hard-wired instead of a trim pot. The fact that the trim pot is there suggests (to me, at least) that there is an assembly step where somebody tweaks the thing to dial it into some sort of range.
That's a fairly common approach to such things -- if you've torn apart much in the way of e-devices (other than computers, and especially RF equipment, 2-way radios, etc.) you'll recall seeing such trim pots in them, usually with a dollop of hot-glue to physically hold them fixed once they've been set.
In fact, the absence of said glue-dollop makes me think that the pot's actual setting isn't all that critical, but that's just a guess.
I'd think that if your contrast doesn't bother you, don't mess with it. And if it does and you aren't worried about opening the case up, go for it.
Of course that's a pretty brutally obvious summation, isn't it?
All of this is just my opinion, of course, so salt to taste.