Well, even with people who write "from the hip" use one form or another of a builder world. Like I said in the definition, it's a place where you build ideas, try out story lines, etc, before they go into the real book. You may not have a cut and dried version like mine, but you very likely have one and simply don't realize it. Mine is always playing out in the back of my mind. My subconscious typically takes control of it and chews on a wide range of ideas (Being ADD has its advantages in a system like this! hehe) in the world until it comes up with something it likes which it then forwards to my conscious mind to use.
Think of it like a foundation, or a "well of ideas" from which your inspiration springs.
Your builder world doesn't need to be quite as complex as mind (my builder world has gotten to be somewhat of a monster, but I like it that way!
), but I bet if you really thought about it, you'd see that you do have one, even if it's rather tiny.
Also, builder worlds are useful for creating and fleshing out a character before you ever start writing. Since characters are the core driving force of most stories, the more you know about a character you'll be writing about before you do it, the easier it is to write the world in which they exist and will interact.
That's just something to think about.