Since EPUBCheck is a Java app, you'll need to install Java. For more information, see
Installing Java on macOS 11 Big Sur.
You could also give the Calibre epub checker a try, which'll also find some common epub problems.
Simply add your book to the Calibre library, right-click it, select Edit book and click Run check. (If the Check book window isn't shown, enable Check book in the View menu.)
If Calibre reports problems, click
Try to correct all fixable errors automatically.