[kovidgoyal]

You're welcome

The improvement comes mainly from changing the algorithm to detect duplicates from an O(n) to an O(1) implementation (using a hash table). And making the detection use multiple processor cores.