The big format that is open for ebooks is epub. Epub3 tries to implement such features as you are interested in.
http://idpf.org/ is the organization that implements it. Then there is readium, which is an implementation of a reader for epub3, that is open source.
For making epubs, there is sigil and the calibre editor. The developers of both are here on mobileread and both have subforums here. Both support plugins, that can be written by anyone.
Another field is converting documents to ebooks. Here on this site, look for example at toxaris word addin. An all purpose converter is pandoc.
There are software projects like scrivener, for writing software etc.