I'd guess it's simply because it tried to use the context to achieve the goal.
Guarding PII is an extra, I guess, and if you want to use a giant cloud-hosted NN to control software on your own device, there are very few default restrictions. And no matter what restrictions there are, when they go beyond prompting, they would have their own ShowHN.
Why is the login information even allowed to touch the context?
For the user's name it's intentional; they template it into the system prompt with no option to disable it.
For 99% of users, the model knows it's in the /home/joesmith/whatever repo and `git log` shows the last change was by `joe.smith@example.com` - trying to keep the user's identity secret from the model would be nigh-impossible so I think anthropic gave up on even trying.
loading story #50386882
loading story #50386677
loading story #50386931
loading story #50387609
I can think of tons of common prompts and tasks for which the AI being easily able to introspect subscriber details would be very useful.
Wish that were itself behind a permission wall, but I guess everyone just uses auto mode anyway so perhaps that ship has sailed.
loading story #50386513
loading story #50386380
Do you use git? I have bad news for you, your email will be all over the place in the context as long as the agent is allowed to read git log, whether it’s injected into the system prompt or not. (Likely retort: but not that email! Sure, but leak potential is obviously still there.)
loading story #50386615
So that if you tell it to do something with your email address it'll know what that is.