You might want to try using
Calibre Companion. Just on general principle, since it is an awesome app for using calibre with android. But since it does not connect over USB but via WiFi, it wouldn't suffer this problem.
Calibre Companion costs $2.99 but it is worth it.