@Aeris: In what mode? USBMS or USBNET?
Regardless, the state of the device itself has no impact either, although if you do turn USBNet off with connections open, those child processes will not die with the daemon (at least with OpenSSH, but dropbear, too, probably).
(That's by design, otherwise, imagine updating the SSHD on a remote server: if rebooting SSHD killed the child processes/connections, and for some reason the update failed, and the updated service failed to come up => screwed).
So, again, best guess: something on your end is keeping connections open
.