Under the main library you have sorts for author, title and most recent. You can also go in and view your file folders. As you mentioned Calibre, that will automatically setup folders based on authors and I think if you play with it, you can setup top level folders for genre, then by author within those folders, or even just dump the files in to the genre folder they belong in.
So if you want to look for a book by genre, you go look under the file option in your library. If you want to do title/author/date sort, go to main library. If you want to do some other kind of setup, load it on to your shelves that way.
Though shelves are broken in that rebooting (and some claim just hooking to your PC) will remove all items from your shelves that are stored on your micro SD card. I have found rebooting does this. Though I only have roughly 150 books loaded on my nook and I am only using about 50% of the unreserved memory space (I have about another 20 books that I don't care to have sorted on to shelves on my micro SD card). So it is going to be awhile before I run out of space to store books that can be effectively sorted on shelves.
It won't lie, it would be nice to have the nook ST setup so that there can be a meta data tag to place it on a certain shelf that way Calibre can automatically set the metadata tag so that new books you load go right to shelves you have setup.
And or a nook application for your Windows/Mac that allows you to manage your shelves and content through your PC without needing something like Calibre (though I do really like Calibre).
It was kind of tedious to go through and create the 9 or 10 shelves I have right now as it meant needing to go through about 20 odd pages of books 9 or 10 times to select the ones I wanted to put on the shelves. Probably only took me 20 minutes to setup though.