[GeoffR]

The font size selected by the slider is scaled by the font size styles set by the publisher in the book's stylesheet, which can be different for different books, even for different books by the same publisher.

You could edit each book and change the publisher's font-size styles before sideloading, but this can be more complicated than it might seem for some books, and there is not really any automated way to do it that works for every book without the risk of spoiling the layout of the book in some way. I think it is easiest just to adjust the size with the device slider for each book (if necessary) the first time it is opened.