Solved, but not with wireguard.
I used tailscale for Kindle:
https://github.com/mitanshu7/tailscale_kual
Not ideal, since I would prefer to avoid tailscale, but it does work for at least the oasis 3 and the voyage. I needed to use
https://github.com/notmarek/kindle-usbnetlite instead of the linked version for it to work on my oasis 3. It otherwise installed and ran.
Note that you do not need to actually run sshd for tailscale to run. You just need usbnetwork installed.
Also note that you need to get the auth key by going into the tailscale admin console, then machine -> add device -> Linux server -> toggle on set up authentication key -> generate install script. Then copy and paste the auth key including the prefix tskey-auth-.
I set up a machine inside my wireguard network as a router for tailscale:
echo 'net.ipv4.ip_forward = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf
echo 'net.ipv6.conf.all.forwarding = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf
sudo sysctl -p /etc/sysctl.d/99-tailscale.conf
sudo tailscale set --advertise-routes=192.0.2.0/24 (ie. my internal network)
And then I authorized that route under the admin console for tailscale, where it pops up with a dialog.
Works with (locally hosted) wallabag, syncthing, calibre connection, so I presume it will work with anything.
GrapheneOS could make this a lot easier, reliable, and secure if they didn't block normal network protocols for no apparent reason.