Go to the customisation for the plugin, and you can set its behaviour. However I think by default the alternate search is set to only check the first format in preferred input order.
It doesn't necessarily have to be a massive PDF, but just PDFs in general will slow it down, by how much depends on the content I think moreso than size. If it has lots of graphics I think that makes it grind rather slowly. There's a few posts in this thread about it if you read back. Now 0.7.53 is out I can start experimenting more seriously with the "first 10 pages/last 5 pages" approach to scanning which hopefully will improve things.