Quick update:
Since turning off the downloading of tags entirely in Calibre (and manually adding my own when needed) the loading time after I disconnect the reader from the computer has gone all the way down to the time it takes me to prepare a cup of coffee before starting to read what was just loaded to the reader. Much much better in other words!