Erm, select a bunch of filenames in the File Browser, right-click, and use the Rename function.
If you use it on multiple files, it will do a bulk rename and ask you to "Rename Files Starting At: _____"
Enter ".xhtml" -- Sigil will recognize it starts with a "." and use the suffix to rename the file extensions instead of the file names.
(I could've sworn this used to be a "Bulk Rename" option...
just plain "Rename" doesn't make it quite as obvious...)
Please do NOT use regex to change the filename links, Sigil knows how to update valid links if you rename files properly
but if you destroy the links and *then** rename the files themselves the links won't match and accidents can happen!!!