Actually, scifi and fantasy are some of the biggest culprits. Check out that wiki article Igor linked to, it described the difference well. That being said... a lot of authors in those genre try very hard to not be, and sometimes they succeed. (And for the record, my favorite genre fiction IS scifi/fantasy.)
Again, I don't mean the term pejoratively and don't have any interest in getting into a "what is art," "what is literature," etc. argument. But there's a big difference between "The Kite Runner" and "Demian" versus Harry Potter or the latest Robert Jordan.