You can specify under the Calibre preferences what fields should be searched. The default setting include the comments which can both slow things down and also lead to unexpected hits.
You can restrict the search to a specific field by preceding it with the column name. For instance something like:
This will be much faster and be less likely to get unexpected results.