You can load a .ui file from a plugin zip file and auto compile it, using PyQT4.uic.compileUI (for an example of doing this look at calibre.gui2.build_forms)
And you can have your plugin tell the calibre preferences dialog to use your custom widget by defining two methods on your plugin
Code:
def config_widget(self):
...
return QWidget
def save_setting(self, takes the widget returned previously and stores the settings)
...
Look at the MetadataSource class for an example (though I would recommen using JSONConfig rather than XMLConfig ina modern plugin.