You click the label "Author(s)". Calibre then sorts the entries on the field Author Sort. Click again to reverse sort.
Names are special when it comes to sorting and displaying. When you display the name of an author on a book you usually want to display it in the normal form "Firstname Middlename Lastname". But at the library (or in Calibre) you usually want to sort the books on the name as if it was in the form "Lastname, Firstname Middlename". Calibre allows you to do just this. It displays the name in the Author(s) field, but sorts on the name in the Author Sort field.
Since the field Author Sort is not visible in the file list, you can sometimes get what appears to be wierd sorting when you press the "Author(s)" heading to sort on name. But if you manually make sure that "Author Sort" is correct for all books, then all will be fine.
Calibre attempts to set "Author Sort" when you import a book the first time, but is easily fooled. So you usually always have to manually update that field to correct the sort order. But once it has been corrected, it stays that way.