Well, there wasn't much (i.e. any) excitement about being able to log in to a shell on the PRS600 via USB, oh well.
Anyway, the "arbitrary code execution" so far has been rather academic. I figured out using the broadsheet driver to update the display, but I couldn't use the buttons or touchscreen, so couldn't write any interesting apps. Well, that's changed now. I can detect button press/release, and at least get touchscreen messages, though I'm not sure what they represent yet.
The "subcpu" appears to be a touchscreen/button controller wired to the second serial port of the iMX system-on-chip that the reader runs on. It's visible as /dev/ttymxc1 and is a standard character device.
The other serial port isn't connected, and there are posts by awesome russians disassembling their readers and getting a console by attaching needles to the first serial port pins. I do this over USB, without needles, and without the need for a login/password, but the idea is similar.
Communications with this controller happen in 8-byte blocks. The first byte of the raw message has bit 8 set, the other 7 do not. The messages are encoded in some weird form that involves either encryption or error detection or something.
Once decoded, you get a message. For the buttons, you get one for a press, and one for a release:
MESSAGE: 03 01 2b 17 00 00 3e 3e
MESSAGE: 03 01 2d 17 00 00 38 38
MESSAGE: 03 01 2b 0b 00 00 22 22
MESSAGE: 03 01 2d 0b 00 00 24 24
MESSAGE: 03 01 2b 0c 00 00 25 25
MESSAGE: 03 01 2d 0c 00 00 23 23
MESSAGE: 03 01 2b 03 00 00 2a 2a
MESSAGE: 03 01 2d 03 00 00 2c 2c
MESSAGE: 03 01 2b 02 00 00 2b 2b
MESSAGE: 03 01 2d 02 00 00 2d 2d
MESSAGE: 03 01 2b 01 00 00 28 28
MESSAGE: 03 01 2d 01 00 00 2e 2e
MESSAGE: 03 01 2b 00 00 00 29 29
MESSAGE: 03 01 2d 00 00 00 2f 2f
MESSAGE: 03 01 2b 63 00 00 4a 4a
MESSAGE: 03 01 2d 63 00 00 4c 4c
I found it kind of silly that three bytes change just to indicate a key release...
EDIT: but then I realised that the last two bytes are just checksums, and that a message only contains six bytes of actual information. D'oh. Time for bed.
Touching the touchscreen releases a flood of messages while you're touching it, and then a "released" message that gets repeated over and over until you send the subcpu an acknowledgement. It turns out that all messages which, when decoded, have bit 8 set in byte 0, should get an ACK. The ACK message is just the original message with two bits changed (which actually doesn't seem to matter, but I followed the firmware on this). To send messages to the subcpu, you need to encode them again.
So, please find attached code and the compiled executable which talks to the subcpu, reliably decodes messages and sends an ACK when necessary, properly encoded. The code is reasonably documented at the points I understand.
The encoding/decoding convolution is interesting, it was a pain to figure out from the disassembly. Now that it's in nice C code, if someone recognises what is might be, I'd love to hear your theory.