I have noticed you can use Darkmode while reading a book. If you open & close the TOC or Annotations after enabling or disabling Darkmode. It's like it needs a refresh?
It still takes 2 taps but it is HEAPS quicker than closing and opening the book again! A few milliseconds at least