Have a look at
GP2X - a linux based portable gaming decice that is COMPLETELY open.
They even includes rudimentary e-book reader in the base software
there is insane number of ported games, console / old computers emulators ...
It can even play divix and xvid files out of box without reencoding.
You receive complete sources plus Software Developement Kit
It runs on two AA batteries, so you can change them on the go.
It has got USB connector (I think active and passive)
It has even S-video connector
It's cheap. Just £119.99
http://www.gbax.com/main.pl