If you find someone who is also a reader, then you always have some common ground for discussion and it's wonderful if you actually meet someone who shares your reading likes and dislikes.
I don't view it as shallow to be turned off by someone who doesn't have any books in their home, though I do think you might have been a bit hasty in your judgement and put a little too much emphasis on the book-owning and not enough on the person - especially since it wasn't a situation where you were making a lifetime commitment
. What if he was dyslexic? I'm more concerned with the person having a sane and balanced worldview, and that can be obtained without being an avid reader of books.