It is probably mostly push-back from the era when people with many disabilities were completely marginalized.

On the other hand, I can think of reasons why it is desirable for accessibility to be a universal:

1. Consideration. Many disabilities would not receive consideration otherwise. Able bodied people rarely think of disabilities unless someone in their life has a disability. Those who do may consider accessibility as an additional expense that their competitors don't have to deal with.

2. Cost. Many products that are developed for people with disabilities are very expensive, thus placing disabled individuals at an economic disadvantage. Products that are developed for the general market tend to be cheaper, both because of competition and because development costs are spread over a larger market.

3. Dignity. Imagine having to go through your life constantly having to ask for assistance because it is difficult if not impossible to find products that meet your needs (or you can't afford those products). It ends up taking a psychological toll.

4. Universal benefits. A lot of those accessibility requirements benefit other people as well. If you need your hands free to do other work, you can use TTS. Wheel chair ramps are useful to people with strollers or temporary injuries. Accessible busses are as useful to people with limited mobility and strollers as they are to people in wheelchairs. The advent of handicapped parking spots led retailers to consider pregnant women and the elderly. Audio signals at crosswalks mean that you can listen for a cue rather than stare at the sign ahead of you.