Agreed, the CSS3 margin spec is quite different.
CSS3 does support paged media and @page, but regarding ePub 2 & 3, the IDPF documentation says:
EPUB 3 defines a profile of CSS based on CSS 2.1 with added modules from CSS3, whereas EPUB 2 was based on a specific subset of CSS 2.
So it looks like ePub3 does not support all of CSS3. Therefore an ePub2/ePub3 validator looks more appropriate than a CSS validator. Alternatively you could use the option button on the W3C validator to use CSS 2.1.
Calibre's style sheets are fully compliant with the ePub specification.