You're not doing anything wrong,
I'm doing something wrong.
The XHTML "Add Semantics" menu is designed to mark one and only one file as belonging to a certain type/category. So if you mark one file as the cover, and then mark another as the cover, the first one is unmarked. This is the preferred behavior since every single one of the types is generally meant to be applied to just one XHTML file.
Every single one
except the "Text" type... which I forgot about.
Report this on the tracker so it gets fixed.