@7hir7een: Huh. That's... not good, I really thought that'd do the trick...
Well, the good news is that stomped on approximately all my current ideas, so, there's that

.
A few non-crash inducing questions: How quick were those unplug/replug (in particular, how long did you wait between the two, and did you notice at which point it froze: after the unplug or the replug?).
And while USBNet is still running, does unplugging/replugging (in itself, without toggling the hack) goes 'splodey, too?
EDIT: I guess that, for now, the best I can do is prevent the switch from happening if the device is unplugged, and restore the same codepath as in r15022 when plugged and hope for the best that the timings are not too finicky to still cause a crash in some circumstances...