Theres a few things that can slow down searches/Calibre in general:
1. Database that has accumulated too much crud over time. Running Library > Library Maintenance > Check Library every so often is good practice.
2. Composite columns. Too many of them, or overly complex ones, or the dreaded
virtual_libraries() function, can slow things down.
3. A failing hard drive. I had one die a few years back. Calibre got pretty laggy shortly before I recognized it.