I agree, the mouse movements is the most visible pitfall.
I have an idea how to fix it, which I hope I may actually make it happen one day.

Have you ever tried magnifying the desktop beyond the screen resolution on Mac (possibly it's the same thing on Windows)? The screen follows the input point, mouse movements. I imagine having a small passive black&white LCD, like 3 - 5 inch, put above or under the Max forming this kind of loupe. This way it would be easy to enter text quickly or navigate the mouse whilst having the nice bigger eInk screen to see everything. I actually already have some RaspberryPi board and a 3 inch passive LCD module for it, the rest is to develop the VNC client for it which would behave like a loupe always following the input point at the center of the screen.