Assuming you used .nef as an example

you can check out the KePub plugins code at
https://github.com/jgoguen/calibre-kobo-driver. I can't remember where exactly, but one of the files there defines the '.kepub' extension so calibre can recognize KePub books made through the conversion plugin.