FWIW all your books could be 1G and as long as the drive can hold them, it would not affect Calibre's speed (the desktop GUI) as the books are in the file system and not the DB.
What can affect:
rogue A/V scanning every file access. <<< White list your Library from
Active scans (leave the manual scan in place)
something affecting disk access (a failing drive, doing multiple read retries. Check the
Event Viewer (Red items involving drives) , overloaded USB hub (external drive)
Lots of metadata changes without running phase 1 of
Library Maintenance to compress old space.
Note the size of Metadata.db before and after running the tool. The DB is what is in memory, not the books