Let us be sure of the terms:
Font is the way a character
Looks (includes the Bold, slant variants)
Font size is how big/little
Failure of the reader to resize:
1) font size is specified in absolute dimensions in the Style
2) What you see is a picture of words (you can't copy/change any word)
3) the reader software has limitatations
There are threads discussing the T1 and using System Local fonts
The solutions are all T1 centric, and make a book into non-inter-changeable device EPUB book .