If I were you, I'd probably respond to someone who said "I won't read your books because you're a man" with "I won't take your money because you're a woman." This is why it's probably a very good thing that I'm not an author, and need not interact with the general public.