if you're trying to validate HTML5 as XHTML 1.1 then it's not going to work.
what you've created is an XHTML5 document (XML + HTML5), and because you likely don't have a <!DOCTYPE html> in the document, W3C is going to guess that it's XHTML 1.1/1.0 because of the <?xml ...> declaration, and throw errors when it encounters HTML5 elements.
W3C validator doesn't auto-detect or allow you to select XHTML5 to validate against, so a way around this is to set the Doctype to HTML5 (Experimental) from the dropdown menu and run it that way, although this will only let you know that you've created a valid (or invalid) HTML5 document. however, if you're using anything related to the epub namespace (ie, <a epub:type="noteref" ...>) it's still not going to pass W3C.
what's more important, i think, is running it through epubcheck to verify that it's a valid epub, since that validator is going to be tailored to the epub specs.
still figuring all this out myself, so if anyone cares to correct me it's appreciated