Indexing doesn't usually slow down reading -- unless the indexer is still processing new books, of course.
Many many books will of course slow down searching through the index.
And collections is a separate database, which also suffers from large size -- one reason I prefer using LibrarianSync for bulk modification of collections.