I used to be quite militant about this distinction

. Then, I read this
little ditty* that brought me down several notches and made me see the light

. (*specifically, pages 21-23 (especially 23) of the link I provided. Very eye-opening).
I realized that the only times I felt compelled to make that distinction was when confronted with
bad science fiction or
bad fantasy. That is not to say that there are no differences at all, just that the differences do not relate to the quality of the book in any way.
Just a thought