See
Does any device support multi-level TOC's? for an example that works as expected in Windows FBReader and OpenInkPot. It's nested TOC is in toc.ncx, for example:
Code:
<navPoint id="708e5056-e9a3-409c-9377-d3f1b00015f1" playOrder="24">
<navLabel>
<text>Genesis Dawn</text>
</navLabel>
<content src="The_Example_split_24.html"/>
<navPoint id="c5b218bb-9241-4994-8905-2c97143cb702" playOrder="25">
<navLabel>
<text>Chapter 21</text>
</navLabel>
<content src="The_Example_split_25.html"/>
</navPoint>
<navPoint id="564c81ab-ccd9-46e1-bffb-5f8abd7c7323" playOrder="26">
<navLabel>
<text>Chapter 22</text>
</navLabel>
<content src="The_Example_split_26.html"/>
</navPoint>
</navPoint>
Where "Genesis Dawn" is at level 1 and contains Chapters 21 and 22 at level 2.
This example seems to have been generated with Calibre, so I'm not sure what is wrong in your case.