@kakapo: it shouldn't (over USB, at least. There might indeed be some trickery needed over WiFi). You might want to check dmesg & co, and/or check that the issue isn't on your host's side (putting the hub in low-power/suspend mode, or that kind of crap. Might want to try a vanilla kernel to make sure it's not doing something funny with the USB stack (but that might just be my inner gentoo nerd sneering over ubuntu ^^).
@geekmaster: Yes, the key exchange process suffers from some performance problems in dropbear since the latest version when libtommath was updated. It's already slightly annoying on a K3, so it must be worse on the slower CPU of the DXG

. I might end up trying a full OpenSSH setup, since I'm already building it for the sftp server, and we aren't really *that* concerned over storage/memory usage anyway.