This epub seems to have an embedded font. So yes, the font is set by the book, but the book also supplies the font. It keeps it in this folder: ../Fonts/ACaslonPro-Regular.otf (which is inside the epub).
If you want to have total control over which fonts you use for reading then indeed you'll have to check every book you read for font specifications. The easiest way to do that is to use Calibre. You can convert each epub you want to read to include all the settings you like, font settings, line height, font (even stripping font settings), margin size etc. You can even convert multiple epubs at a time. I've attached a screenshot of the settings I use so you can get an idea.