Okay. well you are clearly way more awesome than me. That said...
I see no reason why A) a specially compiled X targeting the FAT partition couldn't replace the X offered up in squeeze and B) why
THIS kind of wangle couldn't be used to either convince the native awesome to switch to this mode of operation - or even just bundle another one...
Frankly, excellent work, with a bit of love this could be outstanding.
Hats off to you.
EDIT: I'm having a quick go at build the custom located xorg-server now... fingers crossed.