Steve, wait, have we talked yet about how time travel SF is really fantasy?

Actually reading The Doomsday Book right now and it brings that thought to mind. We have no scientific basis I've seen for actual physical travel (my body goes there) into the past. So why isn't time travel categorized as fantasy?