@theducks - ok, I think I have replicated what you are doing. If I understand correctly you are using the "Save to disc" function to put your books in a folder, because you want the metadata opf file to be there. But you are using the "Connect to folder" function to view what is in that folder, and then using the "Remove matching books from device" feature to delete books from that folder you don't want.
When I do those steps, it does indeed only delete the book format and leave behind a folder containing the .opf file, and the cover.
So I guess the first question is whether this is a bug that Kovid wants to do something about - you could open a ticket and ask I guess. I would presume that you may want two things to happen: (1) when the book gets deleted its opf file (and cover?) get deleted too, and (2) it deletes the folder?
If Kovid doesn't want to change this behaviour, then it would be an easy enough addition to the Quality Check plugin for something like "Clean orphaned folders" which given a root directory looks for folders that only contain .opf/jpg files and deletes them.