To be honest, I don't refuse to buy DRM'd books, but I can see the logic behind it. The way to get a company to change is to arrange things so that the company will make more money if they change. Governments can do this by taxing behaviour they don't like (as long as there isn't a way to avoid the tax). Consumers can do it by boycotting the company and buying from their competitors.
The trouble is that in order for a consumer boycott to work, it needs large numbers of people to join in, and it's difficult to get people to join in.