I'm going to jump in and be the voice of dissent, here. I believe there is great value in keeping the reader in mind even though you are following your muse and writing the book that tugs at your heart.
Printed words do not exist in superposition. They are not like electrons, able to be two different things at once. During the process of writing, the author inevitably comes to points when he or she must decide (a) whether or not to include exposition, (b) how explicitly to render a character's thoughts, (c) which cliches to build off of, and (d) which cliches to defy.
Different groups of readers come to a book with different backgrounds, assumptions, and ambient knowledge about the genre. Hard-core science fiction readers will get bored if you spend three pages explaining the vagaries of superluminal travel. Mainstream readers may be confused if you don't. Older readers are more likely to pick up on references to politics. Younger readers are more likely to laugh at pop culture jokes, and so forth.
So, while you don't want to constrain yourself to writing a book that will appeal only to a narrowly-defined target audience, it can be very helpful to maintain a mental image of your book's typical reader. That way, when you come to those decision-making crossroads, you be consistent in terms of how much exposition to include and which assumptions you make about your reader's mental state. The alternative is to bumble about at random, and that can result in books that are haphazardly tailored to multiple audiences and end up pleasing none.