SF has never been the most popular of genres.
As a rule, the best SF is more demanding of readers than the best of other genres.
More seriously:
At a time when the world is getting increasingly small-minded and reactionary it should be no surprise that the "literature of ideas" isn't growing with the world's population.