There have been significant changes to the way that Apple manages ebooks under OS X 10.9 (Mavericks). Selecting iBooks as your preferred reader application while running calibre under OS X 10.9 (Mavericks) does not work.
At this time, I have no idea if or when this situation might be resolved.
As an alternative to iBooks, consider
Marvin for ePubs, and GoodReader for PDFs, both supported by this plugin.
Marvin has a dedicated
MobileRead forum.
G