[Ransom]

Literary fiction may discard an advancing plot..., specific characters

So may genre fiction. A great example would be any of the myriad stories that take place within dreams, and more specifically, stories that leave you wondering if what you just read was a psychological event; a spiritual event; or a material world event based on the science of many worlds, parallel universes, and hidden dimensions. You'll find most of these in the fantasy and science fiction categories.

or even spelling and syntax... operating by its own rules.

Many genre specific stories do the same, or would really not consider Lord of the Rings to be fantasy? And Tolkien was not alone when it came to making-up his own language.

It requires more effort of interpretation from the reader

This is generally only true because these authors do their very best to use the most obscure words (not "big" words as you misinterpret me) they can find. They dig up old lost and forgotten words that nobody uses today, and do so needlessly more often than not. Why would I use a word like "threadbare" today when almost no one under thirty would know what it meant? when I can be more effective by speaking in modern language and say "wearing thin" or "getting old" etc. Most authors that get pegged with the LF tag are simply poor communicators who come off sounding like a 12-year old trying to impress their English teacher and failing miserably at it. The one and only reason to use an old word is if it is the only one available to convey the meaning you need. As far as obscure words go that aren't old—ditto. I fully expect to come upon obscure words when reading something like Chris Langan's Cognitive Theoretical Model of the Universe where he had to actually invent terms where none existed to convey his thoughts. The same would have been true during the fifties when David Bohm was inventing new concepts like nonlocality and quantum potential. I also expect to see both new and obscure words and terms when reading good sci-fi. (I don't think I ever read an Author C. Clarke novel without learning some new ones.) This simply isn't the case with hardly any literary fiction I know of. Their obscurity is generally just bad writing.

but it can convey thought-structures beyond the means of conventional storytelling.

Name one that I can't find the equivalent of in a genre category.

"Most other people" don't have a definition of literary fiction, but just a vague sense

That's because it doesn't have one except that of being very focussed on words even at the expense of the story. LF is by its very nature incredibly subjective, and most people realize that.

Where most LF authors miss the boat is that they tend to concern themselves more with old and obscure words as a way of making themselves try to appear smarter than they really are when they should be concerned with learning to turn a phrase. The latter, along with having good stories, is exactly what makes a great author a great author. Few were better at it than Mark Twain and GK Chesterton. Consider Twain's describing a child's birthday party as a "pleasant turmoil." No old or obscure words there. Good authors don't need them. Take any number of quotes from Chesterton:

"The true soldier fights not because he hates what is in front of him, but because he loves what is behind him."

"...feminism is mixed up with a muddled idea that women are free when they serve their employers but slaves when they help their husbands."

"It is absurd for the Evolutionist to complain that it is unthinkable for an admittedly unthinkable God to make everything out of nothing, and then pretend that it is more thinkable that nothing should turn itself into everything."

"Impartiality is a pompous name for indifference which is an elegant name for ignorance."

Or how about Ray Chandler:

"To say goodbye is to die a little."

Old and obscure words are almost always an excuse for a lack of talent, and this is exactly why LF doesn't sell. Not because people don't understand it, but because they do.

I've read a considerable amount of literary fiction, classics, general fiction, and genre fiction, as well as having been exposed to a great deal of fiction aspiring to be literary and received instruction on the qualities of literary fiction, in the course of pursuing a Bachelor's Degree of Fine Arts in the English Language.

Hmm—well, like Chris Langan, I ain't had me much educatin. Hadda settle for being what they calls an autodidact. I tries to makes up fir it by being extra special good looking.

"Without education, we are in a horrible and deadly danger of taking educated people seriously." ~ Chesterton

That's all I've got to say on the subject.