Originally Posted by Xenophon
Have you considered computing a hash value for each file? It'd be easy to do while generating, and would save the cost of reading the old one -- you'd only need to read the old hash value. And if you store the hash values in a file of their own, that's only one extra file to read (and re-generate).
We had thought of doing this. It would still have a cost as because of the way files end up being generated it it is probably not easy to do this dynamically so it would involve re-reading generated files to get the hash value generated. It would definitely be faster, though, in that as you say, if the results are saved the original files do not need to be re-read. This is particularly important in the case of a large library stored remotely on a network share (as mine is).