The software does not have to guess anything because the offset is contained in the "metadata" of the pdf file.
The metadata can be changed e.g. with
BeCyPDFMetaEdit (free):
http://www.becyhome.de/download_eng.htm#becypdfmetaedit
A workaround would be to delete the front matter of the file
such that logical page will be equal to the physical page.
(When done with annotations, the front matter could be added again).