Just because I wouldn't have let the lack of books bother me so much, doesn't mean that I think you're a bad person, OP.
We all have our standards and things we look for in a potential mate, FWB, etc.
For me, it's more a sense of humor, kindness, am I attracted to him, and intelligence. There are some very intelligent people who don't read for pleasure.
Actually, you probably did the guy a kindness by not pursuing him.