There is a bug in the custom scheme editor.
If you set the underline mode from something back to none, the editor throws an exception.
If you close the editor, it can not be opened again.
The problem is caused by the statement
in the file tweak_book_gui.json in the style you have changed.
For all other styles without an underline the same statement looks like this:
Changing the statement back into the second one inside the mentioned file by hand, repairs the problem.
Before you attempt to change anything in the file, close calibre and make a backup copy of the file.