The takeway seems to be: don't use $ORIGIN at another time than link-time, or shit happens? And when you do that, maybe the magic -z,origin does is key?
EDIT:.... Nope, the takeaway seems to be that patchelf sometimes breaks stuff, depending on how much mess it has to make of the ELF headers to fit new stuff into

.
Looking at the GH issues, we don't seem to be the only ones hitting these kinds of issues (that, and arm support is not even guaranteed by the author to begin with), so, yay?.
Setting a couple of bogus rpath w/ different lengths seems to shake things loose and leave you with a working? binary in the end, but the first (few?) set-rpath call(s) on a binary with NO existing RPATH/RUNPATH entries is ... not always sane.
Potential issues may not be limited to this, though, so, YMMV.