Apparently it doesn't happen that often, because if the publisher paid a sizable advance, they would get a script doctor in to get the manuscript in a publishable condition. Here was one high-profile case where it did happen: Emma McLaughlin and Nicola Kraus, authors of the Nanny Diaries, had a contract and advance with Random House and when they received the manuscript, Random House said no.
http://observer.com/2004/02/why-naug...-random-house/
I don't know what happened to any part of the advance that was paid out, but I never heard of a lawsuit, so I'm guessing they wrote it off. These two authors did go on to publish more books, but not with Random House.
eP