Right, The Enemy is the true prequel -- I assume since it's the only book in the series in which Reacher is still serving (rank, Major, Army MP). The others are all mystery/suspense novels dealing with his life subsequent to his having left the military and his adventures therein.
However, The Affair tells the story of the particular events that led to his leaving the military and since that's mentioned so often in all of the novels, the full story of how it happened is beneficial to the reader (IMO).
You're probably right, though. If I was just starting the series, I guess I'd probably read The Enemy first and then The Affair.