I'd suggest to use ssh keys inside the secure enclave on macOS that require your fingerprint for every new session [0]. That way agents cannot initiate anything, and you will always be aware when a new ssh connection is being made.
[0]: https://gist.github.com/arianvp/5f59f1783e3eaf1a2d4cd8e952bb...
Or in a Yubikey and set the touch-required mechanism (light in the key will blink when it needs a touch to continue the auth).
loading story #50426013
loading story #50426068