You should reevaluate if you can instead use multiple public keys where you need to connect.

It’s a mind shift worth considering.

You then can accept that your ssh keys are different per hardware token (because the private key cannot be extracted).

No more sharing of private keys in your password manager.