You have to manually start the change, it doesn't do it continuously.

For the vast majority of services, even if this action fails and the wrong password is saved (!?) you're still just a "forgot password" click away.