This is an issue that's been keeping me busy for years. I absolutely cannot understand why Apple wouldn't fix the issues mentioned. I get that they use their PDF engine for a lot of stuff and it is really great to have as a printing option etc. but it does indeed mess with Adobe Clearscan OCR and other types of files that need a lot of recalculation to fit into Apple's PDF specifications (which then ruins OCR that does not fit and blows up file sizes)...
There's been mention of Skim here which is a great app unless you want to use it in conjunction with a tablet or other reading app (e.g. on Windows). The only option to get files out of it again complete with annotations is via ... the OSX PDF export which will then ruin OCR and possibly increase file size for those PDFs that it couldn't handle before you started using Skim. This is true of most of the PDF readers on the app store as well including the otherwise fantastic “Highlights” app (the creators know of the issue but chose to treat it like Apple does
)
Then there's of course Adobe reader which isn't too bad (non destructive, honors encoding) but clunky in the annotations department. There's also a Foxit Reader for OSX which is a bit zippier and feels more lightweight, has tabs and context-menu annotation but no annotations overview.
And, this has also been mentioned already, there's PDF Expert, which is the app I have been using ever since it came out. It's fast, has tabs, good annotation features, gets regular updates, has annotations overview and a great thumbnail view for easy navigation. It has a steep price which I think is worth it (and it is on sale from time to same) and has a demo that gives you full functionality for 7 days...
(I also have 1 spare promo code still somewhere and would share it with you, gives you a 66% discount; and for the record I am in no way affiliated with Readdle, I just regularly pestering their support with inquiries and suggestions and they have been sending little thank yous from time to time)