Did a quick web search and found this bug report from 2016.
https://discussions.apple.com/thread/7419388
It had no answer. It appears to be exactly the same bug you are asking about. So my guess is this is a very long standing bug in how Books builds its toc from the NAV that Apple has no interest in fixing.
You might try simplifying it to be a 2-level level or just a 1-level NAV, then create a non-NAV xhtml table of contents to show the user that is more complete.