@BetterRed: My Library is on a USB 3.0 external drive, so that's not the issue. You are right that add times are affected by duplicates though. The custom column this is the reason though, see my response to kovidgoyal below.
@kovidgoyal: thanks for responding to my crazy thread attempting to test the limits of your program!
Regarding your first suggestion, I add metadata from goodreads, so the book descriptions are not long, usually a paragraph or two at the most.
Your second suggestion is exactly it though! The reason search was choking is because of custom columns built from other columns, I changed the setting you mentioned and search was snappy again, even with ~30,000 books in the library, most of which are duplicates. Amazing!
Thanks for entertaining my idea and helping me find a way to improve search in general!