Isn’t the macOS keychain user-readable by default?
Assuming the attacker has user rights (which it needs to read the key) it could also read the passphrase, right?
Update: I tried it out and stand corrected. ssh-add doesn’t store it in the user-readable keychain. So the passphrase is protected. (And also not recoverable by the user).