[Prestidigitweeze]

In terms of writing (rather than casual speech), Robert Graves argued that it is precisely because English is imperfect that it requires logical usage on the part of the writer. English doesn't have the purified palette or grid of perfect constructions found in French. But what English loses in perfection, it gains in expressiveness and richness.

While the survival of words does depend on popular consensus, the person who wants to be understood will make use of every advantage which helps the reader to understand, and logic is one of them. For that reason and not mere consensus, the writer will choose to avoid the phrase could care less.

The other reason to avoid the phrase is because it's distracting. The controversy it has inspired on this thread alone should be enough to show that using it can take the listener/reader completely off course whether you feel that person is being pedantic or not.