The <title>...</title> tags are not used by Calibre when defining chapters/TOC levels.
There are (at least) 2 ways to get the correct TOC nesting.
- The first is to find out exactly how the source document has defined each nest level in the XHTML files. It could be as simple as <h1> for level 1, <h2> for level 2 and <h3> for level 3... however you may not get that lucky! You can try opening the CHM in the Calibre Viewer then using the right-click Inspect feature. Once you have figured out exactly how the chapter headings have been defined you can use the info to construct the correct entries for the box Convert - StructureDetection - Detect chapters at and also for the 3 TOC level boxes on the Convert - TOC page. This method requires you to have a working knowledge of html.
- The second approach requires less technical ability. If you have done your CHM to EPUB conversion and you have a correct, but flat, TOC you could try this EpubFixer utility to massage your TOC to the appropriate nested structure. It's all done using drag-and-drop actions. The utility does more than just TOC editing but I personally don't use it for anything else.
If your CHM uses more than 3 nested levels I don't think you could use option 1. On the face of it option 2 seems to work but I don't know whether the various ereaders support nesting levels of 4 or more.