Here's a video showcasing the issue:
https://streamable.com/hciuri
Using Sabon typeface, I'm on page 82 of Siddhartha, after changing to another book and returning back to Siddhartha it jumps to page 94.
Now if I get rid of custom fonts, this behavior doesn't occur.