From what I've seen, people seem to think "literary" means "good writing", but actually it's a style of writing and there is a lot of bad literary fiction.
From Wikipedia:
Literary fiction is a term that came into common usage during the early 1960s. The term is principally used to distinguish "serious fiction" which is a work that claims to hold literary merit, in comparison from genre fiction and popular fiction (i.e., paraliterature).
In broad terms, literary fiction focuses more upon style, psychological depth, and character. This is in contrast to Mainstream commercial fiction, which focuses more on narrative and plot.
http://en.wikipedia.org/wiki/Literary_fiction
As with all genres and styles the borders between them are vague and ambiguous.