Igor nailed it... specifically genre fiction refers to fiction that has a genre.. Fantasy, SciFi, Mystery, etc. Usually it is formulaic and may deviate here or there, but will follow a certain pattern so with fantasy you may have hero's journey, or the hard boiled detective in mystery, etc. This is in contrast with literary fiction which doesn't necessarily follow any set pattern, may cross many genre lines, and is usually much more metaphorical. If they were movies, "Iron Man" would be a genre film, and "There Will Be Blood" would be literary fiction.
I don't use "genre fiction" as a knock, either. They're different things, and do different things.
I mainly read nonfiction (no time for fun books!) and when I read fiction I tend to read literary stuff, but lately the old fiction itch has been going nuts and I just don't want any Dostoevsky.