I understand you as saying that you want the ability to specify that a single-valued column's value be unique in the database. There is nothing in base calibre that will do such a check.
It can be done in the
Action Chains plugin using a chain that checks for duplicates when a book record is modified, a process that will probably be slow but has the advantage of not needing any programming. Ask in that thread if you want to try this and need help.
Best would be to write a custom plugin that uses calibre's metadata_changed event and cache.get_item_id(). The plugin would check if a value (an 'item_name') is already in use and if so then show a warning. It could mark the books. I think this can only flag the change, not prevent it from being made. I also wonder what would happen if the change is being made by something like bulk metadata edit. My guess is that you would need to write the plugin, as no one who does plugins has needed/wanted this ability.