Well, I got Xfbdev running. I had to patch it, however. Patch is attached, buildroot will apply it automatically when you put it (gunzipped) in the package/x11r7/xserver_xorg-server directory.
I also needed to patch the resulting binaries since I did not want to create a chroot or mess with my root partition. Thus I patched (in Xfbdev and xkbcomp) as follows:
sed -i 's/\/usr\/share/\/mnt\/us\/sh/'
sed -i 's/\/usr\/bin/\/mnt\/us\//'
(with GNU sed on the host machine)
That said - it works resonably well with a big glitch. I didn't test input on the K3, however - I used x2x for a quick drive.
The big glitch is that it gets the ordering within framebuffer bytes wrong. So it will display odd/even horizontal position exactly the wrong way - stuff ends up garbled. Found traces of others having the same problem on the Web, but did not dig further.
Well, and then of course it's not updating the eink screen either. For testing, I just used a quick and dirty loop like this:
while sleep 5; do echo 1 > /proc/eink_fb/update_display ; done