A new development happened. I used my bluetooth for the first time today for an audiobook (I rarely listen to them, first time with the PW4). I have not connected to Wi-Fi today, but apparently connecting to the bluetooth bumped the books up again.
Trying to check the times now on the device and books, hopefully that fixes it.