I just tried yesterday's release 7.21.0, and
now it works!

So, whatever you did, it may have made the difference for me.
For good measure I tried to play audio from the browser first and simultaneously, but even that did not break the playback again. Hopefully it will stay that way.
One thing I did notice is that the highlight for Read Aloud when the audio is coming from the book itself is not following the highlight color from the book.
In the .opf there is a line like this
Code:
<meta property="media:active-class">-epub-media-overlay-active</meta>
Which indicates that in the .css the following attribute can be used to indicate the highlight color.
Code:
/* Media Overlay highlight color: yellow */
.-epub-media-overlay-active {
background-color: #FFFF00;
}
Is this something that on Calibre's road map to support?
If not (soon), is there a way to configure the highlight color for Read Aloud through custom styling?