It's really more the economic impacts...all the money governments have spent in particular since 9/11 in particular as well as the time and attention devoted to "fighting" terrorism. It takes away from what might have been done/accomplished otherwise.
Maybe we should look at it like insurance companies do. Determine what an acceptable level of risk is and implement only what is needed.
