Have you converted your books to one of the formats the Nook can handle (e.g. ePub)?
When generating the catalogs did you set the compatibility mode to Trook?
Note also that there is now a dedicated
calibre2opds forum that has just recently been set up.
When he sees this message Dave Pierron might have some other ideas as he owns a Nook.