I think you are right - it's harder to "pick" literary non-fiction than fiction and of course poetry. I felt pretty confident about Robert Graves' book because he was clearly one of the important English writers of the 20th century - a great poet, but also a fine writer of novels and a number of non-fiction works. I think Barbara Tuchman is a fine historian, but apart from those examples, I would really have to think quite hard to come up with a good list of literary non-fiction. Like you caleb, that probably means I don't read anything like as much non-fiction as fiction!
That book by Hilaire Belloc sounds interesting, fantasy fan. Hold that thought for next time!