Well, that's the trick of it. Some hardware functions are typically run by drivers which are not open source, and others might only be available as "binary blobs". Also, while directly linked software is covered under open source licences, other software on the device (like, all the utilities, etc.) are not necessarily open source.
Simply because
some components you need are open source doesn't mean that you can necessarily run the OS off the open source components. For some devices, the work to make that happen has been done - primarily in routers, where you have DD-WRT, OpenWRT and several other open OS's, but on a range of devices - take
Open Inkpot, who use these forums.
TiVO use a trick of releasing the source code, but implementing hardware check for "approved" firmware, or it won't load, this is known as
Tivoization, and was one reason for the GPLv3 replacing v2 - as more and more projects shift to v3, that trick becomes less and less viable.