If cover.jpg already exists in the .opf file AND is referenced somewhere in the initial html/xhtml file then replacing it should work, though in some cases the dimensions of the original file may be referenced, so it might look weird if your new cover.jpg is different dimensions.
Calibre won't see the replaced cover unless you ask Calibre to re-update it's metadata based on the actual file.
If the file doesn't already exist and you just stick a cover.jpg file in it won't do anything. Technically you'd be creating an illegal epub, but most readers will just ignore the file.
Sigil is another option if you just want to edit the existing file without converting it.
Changing the cover using the Tweak epub feature does change the original file, whereas just updating the metadata using Calibre's GUI does not - at least not until you re-convert, use a save-to-disk feature, or send it to your reader - at that point the epub will be updated with the latest metadata Calibre has.