Since the deptree is so insane, I'd go with the latest buildroot release that supported gst-0 (I already forgot the tag ^^).
(Build, then) feed it the PW2 TC from
koxtoolchain.
c.f.,
https://github.com/koreader/koreader...2020.02-config for an (unrelated) example of using a Kindle TC w/ BR.
Then, either you fudge the BR package directly to build the right gst version, or you build it manually inside BR's sysroot (after having let BR build the full deptree for the "new" version).
----
EDIT: To be clear: that's the "cross-compilation" approach. Doing it on-device wouldn't be any fun (but you'd essentially need to chroot into a real distro in there to do it), both for the CPU, the storage, your sanity, and it'd take a ridiculous amount of time to build stuff.
Same general idea on the Pi, with the same constraints, if not worse on a zero. You just gain possibly not having to deal with a chroot. And you'd probably have to deal with most of the same quirks as if actually cross-compiling, so, eh.
(Yeah, I'm the wrong guy to ask about the "build it on arm" approaches).