Take a rubber shotgun that looks real, point it at him tell him to get on the floor and give you his wallet, after he does so, tell him he passed the test on how to properly react to a mugging. Congratulate him and charge him a fee for the test (Remember you have his wallet )

. In all seriousness you definitely have right to be annoyed.