This is not something I have personally thought about a lot. Perhaps I'm too young to fear bit rot
Certainly, I have so far never lost any data (that I know of) to bit rot. So I can't contribute much to this discussion.
Simply maintaining a hash which is updated on legitimate operations is fairly trivial to do. The question is really what to do with that hash. ldolse's suggestion is one worthwhile use, though better, IMO, would be to add support for declaring a particular format as the "Master format". calibre would then ask for confirmation before running a conversion that would overwrite it.
As far as combating bitrot is concerned, I don't really see how the hash would help. After all, say calibre tells you that the file has changed (this would only happen if you ask calibre to check, for example during a db integrity check). Then what? The file has already "rotted" not much calibre can do about it. I suppose you could then go into your backups to try to find a pre-rotted version.