Assuming you have all the dependencies properly installed, see
the INSTALL file.
If you play around with the commands revealed by python2 setup.py --help, particularly osx32_freeze, you might have some success... then again, you might not. calibre does not support building from source on OSX and Windows, so you are kind of on your own.
Why do you need to create a DMG? There is already an official one.
If you want to make changes, you should submit them to the project, rather than redistribute a modified version that has less of an audience...