You couldn't have liked him much if that was all that it took for you to make such a sharp exit.
Personally, I'd be far more concerned if he hated dogs or cats than if he didn't read. Non-reader would probably be on the list somewhere though.... if I actually had a list that is.
In saying that, I have an uncle who doesn't like reading at all (his wife is an avid reader), but he's a charming and intelligent guy and he and his wife are 30-years happily married.
Edit for a follow-up to wyndslash's post, a guy who considered himself better than everyone else because they are 'well-read' is possibly more of a turn-off than someone who doesn't read. Arrogance, superiority, they would definitely be on my list!