The following is: a) speculation, and b) is from the perspective of the GUI, but as you no doubt realise there's a lot of code sharing between the GUI library manager and the command line tools.
When a book's metadata is created/updated the new data is written to the relevant database tables and and to the metadata.opf file in the book folder. If the changes are made via the metadata edit dialogues the writes to the database tables and the write of the metadata.opf file** is done when the OK button is pressed.
But, if you edit in the book list using the F2 key and Tab I'm pretty sure the writes are done for each cell edited - so if you press F2 on Rating, enter a value and press Tab, the books rating value is written to the database and a fresh metadata.opf file is written; then if you enter some Tags and press Tab, the Tag values are written to the database and a fresh metadata.opf file is written… so on and so forth. I'm wondering if the calibredb set-metadata command works in a similar way.
** metadata.opf writes
Metadata value alterations can cascade though many books, e.g. a spelling correction of 'Vapmires' to Vampires' will generate one write to the database, but every book tagged with 'Vapmires' will get a fresh metadata.opf file with the correct tag value - i.e Vampires'.
To avoid long delays waiting for many fresh metadata.opf files to be written, fresh metadata.opf files are always written in a 'slow' background thread (some call them 'lazy writes'). On my rig they chug along at about one a second.
IIRC the calibredb backup_metadata command (without the '--all' option) will flush through any outstanding writes in the queue - might be worth giving it a try.
BR