The size of a book is irrelevant, since all calibre has is a reference that a file is there (plus a one-time recorded size).
And the number of records is not one of the big slowdown factors -- some people have many many thousands of books without being noticeably slowed down.
Displaying many columns in the Tag Browser is one thing that can slow calibre down.
Fortunately you say you only have one custom column created from other columns -- those are also heavy since they are dynamically calculated.
Might also want to see if you can trim down the column icons.
...
Most of these things are all right in moderation -- and useful -- but the thing is, they build up and cumulatively slow calibre down.