There are two different options there.
Remove spacing will remove the lines at a scene break, too, and Insert blank line will add them between every paragraph.
What you want to do is just leave both unchecked; then calibre won't make any changes there.