Metadata fetching speed is a function of - server responsiveness, network performance/congestion, volume of data, number of sources - not much the calibre s/w can do about any of those, you could try reducing number of sources. The first two can vary significantly over time - i.e. today it might take 2 minutes tomorrow 5 minutes, not unlike commute times in large cities.
If you want to download metadata for many books use the Bulk MD facility, it runs as a background job, whilst its chugging away you can do something else in calibre or elsewhere - even have sleep
Psst, I don't do many MD downloads, but when I do I use Bulk even for one book - there's always something else to do.
BR