What I did to fix this, was to:
- Start a conversion
- Select "None" in "Chapter Mark" (the original setting was "Pagebreak")
- Do the conversion
The resulting EPUB looks ok in Calibre's EPUB reader, the TOC points to the correct places and contains meaningful entries for the real chapters. The resulting epub also have a size of 0.6MB, instead of 3MB.