I think this could actually enhance the experience if done right, with the right books. As an example, I used to read fantasy books by an author who liked to make musical allusions in the story, something a character played or wrote usually. One book even had some of the pieces a character wrote written out in the back of the book. This is the kind of thing I could see benefiting from embedded audio clips or the like. I can't think of anything I've read that would benefit from video in a similar way, but I wouldn't be surprised if it existed. (Just to note, if this existed I would argue that it should be entirely optional, perhaps with "traditional" versions and "enhanced" versions.)
Now, what I could really see taking off is a smellobook or a tastybook.