[Quoth]

DOS had
CON: — console (input and output)
AUX: — an auxiliary device. In CP/M 1 and 2, PIP used PUN: (paper tape punch) and RDR: (paper tape reader) instead of AUX:
LST: — list output device, usually the printer
PRN: — as LST:, but lines were numbered, tabs expanded and form feeds added every 60 lines
NUL: — null device, akin to /dev/null
EOF: — input device that produced end-of-file characters, ASCII 0x1A
INP: — custom input device, by default the same as EOF:
OUT: — custom output device, by default the same as NUL:

DOS was a copy of CP/M-86

So certainly some of those existed on DOS and caused problems on Win9x (it was really a sort of shell on DOS 7), so with NT 4.0 to Windows 10, you probably can't create folders (directories) with those names.

I just tried on my Win10 VM these: CON, PRN, AUX for folder names and got invalid device error, and name reverted to "New folder"

I suspect on Windows that con.txt is also invalid.
DOS and Non-NT Windows had COM1: to COM4:, NT has 256 Com ports (Win2K, XP, Vista, Win7, Win8, win10 & Win11 are NT 5.x and later). Also LPT1 to LPT3, don't know how many NT has.
So not just A: to Z: is illegal in folder or file names on Windows.