I personally get a nice hit of superiority for every book I read. That's why I love the Deathlands novels and Resident Evil movie-tie in books -- I can whip through a couple a day and that makes me obviously so much better than the rest of the plebs.
OK, seriously, I LOVE to read. But sure I know lots of non-readers. One of them watches documentaries and the Discover Channel and the History Channel practically non-stop... while the last book I read was a apocalyptic gun fantasy with magic mutants. And I spend way too much time blogging about Twilight and every little thing I think about it. So clearly "reading" isn't necessarily better intellectually than "not-reading".
So I don't consider there to be any reason to get angry at my non-reading friends.