You *can* remove one book from the Virtual Library.
Just remember -- a Virtual Library is a
search, not a
list.
So... tack on another parameter to the search, a logical NOT operator that matches only that book. By id#.
As for bad manipulation destroying a selection, perhaps using Mark Books can help.

There is a toolbar button you can add, and the default shortcut is CTRL+M
Select a few books, CTRL+M to "Mark" them, and build a list of marked books. Then add the marked books to a tag.
Mark Books -- one of the more obscure features of calibre, which no one seems to think about much. But I find it quite useful.