Yes, an unknown character from bad coding. Others that I have seen fairly frequently are:
(well I pasted in a few, but they were translated and did not show as pasted!)
The black diamond in your case is likely an m-dash, and you can do search and replace to fix it.
But I also do not understand why they sometimes show in the editor panel and other times not -- and why getting rid of a <span> or <div> can make them appear. Anybody know?