Because it's just one more part to get junk into?
I've seen more empty and unused SD card slots than used ones. On my phone, my slot is actually wobbly and probably will eventually fail on me. There are too many sizes, too many different reads and writes, and other specifications that it's pretty confusing to buy. Not to mention that they are just NOW coming down in price.
Android doesn't take advantage of it very well, and Apple would probably have just as hard of a time with it too.
The other issue would be, like in my case, that the slot would break eventually.