I have a "formula" in my head, but I am having a hard time figuring out how to express it, so I will give an example: Geek Love, by Katherine Dunn. It is about a very extreme family of circus freaks. Much of the plot is fantastic (as in not realistic), and yet I completely *believed* the characters. Yes, if I were the least-freakish child of circus freaks, I WOULD cry myself to sleep at night for being "too normal." I found this completely consistent (and interesting) within the world the author set up.
So I guess my formula is: take an unbelievable setting, and make it believable by populating it with believable characters. And yet, I checked "realistic fiction" in the poll, that is my main reading. I think the formula can apply there too. I recently read and loved "American Wife" which is a fiction book based on the life of Laura Bush. I knew almost nothing about Laura Bush going into the book. I found the character that Curtis Sittenfeld created in her book to be entirely believable in her choices. So now the formula starts with a niche world rather than an unbelievable world. Of course, the setting and characters must be interesting to me, too, but this idea of "do I believe the characters?" goes a long way toward whether I love the book or not.