On reflection and after peeking in the library database I'm wondering if the presence of duplicates would affect search times - might affect add times.
Which file system and what sort of drive are you using - if its FAT on USB2.0 I would expect that to be considerably slower. If its a network device then you'd better read this ==>>
I am getting errors with my calibre library on a networked drive/NAS?
BR