I see. I don't tend to switch that quickly, I was just testing it.
I now sync manually, exit the book to the main book menu and sync again for all devices. Seems to be working better.
I also re-enabled saving of annotations on the iPhone app, not sure if thats related to it (probably not).