OK, I do not use this functionality, so I do not know how it behaves in real life.
If it is like you say, something should be done. As it depends on type of graphics on page, it might be related to Adobe SDK. The page has to be rendered and different pages might consume different amount of RAM. When anotation is invoked and there is more than certain amount of RAM needed for the task, swaping of RAM to "disk" must be done. It takes time.... And two processes might run and "compete" for RAM and inforce massive swapping of RAM there and forth. It could be something like this, but of course it is only my wild guess.
Maybe that a new pdf reader will behave better....