Thanks chaley (and theDucks for their feedback). Those tips above look promising, obviously calibre's underlying apis have changed a bit in the last 10 years unsurprisingly to add this new_api stuff which looks to be worthy of some investigation.
I did find one other performance bottleneck which is the set_marked_ids when it is called with 50k rows (taking 11 seconds on my machine). I was progressively adding print statements through calibre source to find out where the slowdown is, got as far as refresh_ids() being invoked by marked_changed().
So my plugins normally do these two pairs of calls, for example:
Code:
self.gui.current_db.set_marked_ids(marked_ids)
self.gui.search.set_search_string('marked:library_duplicate')
In this situation because I am issuing a search straight after calling set_marked_ids I wonder if there is some expensive part of this "refresh" going on that doesn't actually need to happen because of the search afterwards. Or maybe it is that refresh which makes the search so fast of course...