Calibres speed is mostly governed by the size of Metadata.db and RAM
Loads of custom fields, long winded comments will make the DB bigger.
The actual size of the book only matters when converting, editing, viewing and embedding metadata.
Do run the first phase of Library Maintenance if you have done some major pruning / renaming. Those entries are only inactive until the house is cleaned
