After having played around with it a bit more, I think it has something to do with opening the PDF before adding it to the library. I tried a couple more PDFs but added them to the library first and highlighting seems to be ok. I'm going to retry on the previous PDFs I had problems with. How so I remove something from the library so I re-add it later?