I'm guessing it's because some much of the stuff on TV is tripe and you have only so many channels to choose from. With books, you always get to choose. If you want a quality read, you can choose a quality read. If you want pure entertainment, you can choose pure entertainment.

I've found TV much less disdainful since tuning into online streaming and library DVDs because of that. (For fun, I decided to hook up my TV to my apartment's cable connection and discovered that the landlord provides basic cable. Ten minutes later, I moved the TV away from the cable jack and back to the ADSL modem. There just wasn't a point to leaving it on cable.)