@hobnail Your dream has (nearly) come true!
Well this was quite long. I'll post some screenshots here as well as in the main post, but I'll explain it all in detail tomorrow. It's nearly 1 A.M. here and I really need some sleep.
So what I did (yes I know, it's not the best technique) was to use Xvfb (fake video driver) to create a "fake" display (:1) and then capture a screenshot of the display with ImageMagick. After that, displaying it continuously with FBInk. No touch for now, but you can input things with xdotool on the SSH client.
It runs Fluxbox and I've installed XTerm and Firefox, worked quite well.
I'll continue that tomorrow !