IIRC, I stopped playing with rmmod/insmod since the K4 w/ USBNet for various weird reasons (and instead ask the framework to do it for me, here, via volumd).
The usbnet script probably still have my ramblings on that subject in the comments.
Looking into that very thing with the introduction of OTG was the incentive for the PW3 @knc1 is currently tweaking for me, so, TBC
.