I'm also pretty sure that most dedicated SF readers could point out entire sub-genres that they don't like since it is an extremely broad category that caters to a wide range of tastes.

The quality of most SF literature depends upon the background of the reader. In my case, I get a lot out of stories that speculate about our near future. These are books that ask about the social consequences of things like biotechnology and artificial intelligence. Sometimes they deal with interpersonal relationships, but the scope is usually society. I'm okay with the latter because I frequently think about the consequences of actions on society at large. On the other hand, I don't get very much out of space operas. This doesn't mean that space operas are bad. It simply means that I don't have the background to appreciate them. Simply put, I don't have to tools to derive meaning out of war nor do I find pleasure reading about war. That being said, there are people who can. Why should I dismiss their tastes because of my own shortcomings?

All of this applies to other forms of literature as well. In order to get something out of a book, may it be intellectual or emotional, you have to be equipped to understand it. SF&F are the most obvious cases because they deal with things that do not exist. Yet other forms of literature are no different. To enjoy a book, it helps if you can relate to the protagonist, understand the cultural context, and know the trappings of Literature. (If you ever want to get a chuckle out of high school students who are studying the Great Gatsby, you simply have to say, "Sometimes, just sometimes, grass is green.")