Kindle for Mac *can* display books purchased from 3rd party vendors. All you have to do is open the file with K4Mac to add the file to your bookshelf. It's still pretty useless as a library manager, since it doesn't have any organization features. Also, K4Mac files can't be read on your Kindle (the DRM is device/app specific), so it's impossible to sync files with Kindle.
I've not experienced the behavior you're seeing with Calibre, but my guess is that the file has different metadata than the library entry so, Calibre thinks it's new. Try opening it from within the application instead of via Finder or, if it's DRM-free, try converting the file (even to the same format) and see if that helps. Calibre can't always read the metadata on Topaz books (e.g. K4Mac Topaz files download with the .azw extension and Calibre can't find the metadata, but if you change the extension to .tpz, Calibre knows what to look for).
Amazon is unlikely to ever release a library manger (à la Sony Reader Library), Amazon's library management and backup is all online. Besides the prominent "Manage Your Kindle" link in the Kindle Store, there are:
http://kindle.amazon.com/home/welcome
and
https://www.amazon.com/gp/ays/index.html?app=home