Perhaps I'm misunderstanding the question, but I think this is what you want. Basically, you are using the wrong feature to perform the operation you want. Step by step instructions are shown below.
- Highlight each book individually, in the exact order the books should be numbered. (Do not do a bulk selection / select all. They must be done singly and in order for the numbering to be automated correctly!)
- Click on the small arrow to the right of the Edit Metadata menu button.
- Select Edit Metadata in Bulk in the menu that drops down.
- The Edit Meta Information dialog box will appear.
- Remain on the first tab, called Basic Metadata.
- The last field you can type text into is Series, enter the series name.
- Underneath the Series field are two check boxes.
- Place a checkmark in Automatically Number.
- The checkbox to the right of Automatically Number will become available.
- Place a checkmark in Force Starting Number.
- The numeric field to the right of Force Starting Number will become available.
- Place your first whole number in this field. (Note: I'm using Calibre v7.32 currently, only whole numbers - without a decimal / fraction - are supported. Possibly a bug since series_index supports decimals. Calibre puts the [n] part of the series information into an internal database field called series_index.)
The first step cannot be stressed enough. If you select books in the wrong order the series info will be renamed and renumbered accordingly. So Book1 might become Book9 if you do a multiple selection/select all.