[Manichean]

OK, so Calibre seems to genuinely have hung itself- I was asking because people often assume that operations taking longer than they anticipated have frozen, which isn't the case most of the time.

I don't think you can customize the internal metadata reading short of editing the source code, but in the add books- preferences page, you should be able to select what happens when Calibre believes it has found a duplicate. One of those options should allow you to add the book files anyway.

The accepted method seems to be to add larger batches of files in smaller chunks of, say, some 2000 files or so. Such a chunk tends to get added all in one. All considered, that shouldn't be too difficult with your situation and would probably be the easiest way.

EDIT: I just realized I overlooked that you already did smaller chunks. Disregard the last paragraph, then.