It seems as if the NC is sorting incorrectly. I can imagine several classes of bugs that would make it do that, but I have no clue if the bugs really exist. One type of bug would cause it to fail once the number of books goes over some magic number. Another would cause failure when the amount of metadata goes over some value. A third could be caused by a corrupted internal DB (assuming the NC has such a thing).
The only way I can see to debug this sort of problem is to factory-reset the NC. The reason for the factory reset is to remove anything that the NC might be remembering. Next, set the plugboard the way you want, then start adding books. Add them in small quantities and check if the sort stays stable and correct. Don't change the plugboard in the middle of the experiment. It might be best to start with the books that are currently failing.
A factory reset will disconnect the NC from your B&N account. Be sure that you can reconnect it before you do the reset. I don't know what "rules" B&N imposes for device counts, or whether B&N will recognize that the same device is reconnecting.