AFAIK (I implemented the catalog feature in calibre) it's not possible to directly hyperlink from within a book to another book. Apparently that's only possible in Java from the shell. However, within the catalog, you can type the name of a book, search My Items, displaying a link to the book, which you can then open. So it's possible to open another book with a couple of steps.
Here's a link to a posting in the Calibre forum describing the catalog feature.
G