Examples of the submenu problem are the plugins "View Manager" and "GoodReads". Clicking on the button opens a dropdown menu under the button. Similar with your plugin button if clicking into the area where the down arrow is situated.
See picture 1-3 of
https://imgur.com/a/OFwjVtO
Only when users choose the configuration menu from the dropdown menu, it opens another window where you can change values. Whenever this window is open, you can open another window. An example you can see in picture 4 of
https://imgur.com/a/OFwjVtO: Clicking on the "Edit 'Shelf Add' Action…" button in the Goodreads configuration window. Closing the "Edit 'Shelf Add' Action…" window closes only this window.
-----
A solution for your plugin could be to open just a window for configuration issues. Maybe you could also spare the submenu "Formatting" and "Other" by putting the "Other" choice as a button into the formatting option window and to open an overlaid window for these choices. (Similar as "View Manager" and "GoodReads" do this with their other buttons inside the main configuration window.)