It's the nature of the beast and all my research shows it can't be solved fully. Just like human social engineering.

The only 100% safe solution is to not reveal the private info in the context. That's exactly what anthropic is doing here: removing the logged in user email from the system prompt.

Make no mistake though, that's a weak fix. The email is usually just a "git log" away, and for many tasks it finds its way into the context quickly.

Internet access must be blocked for agents dealing with private information. This makes agents vastly more cumbersome to develop or much less useful.

So the frontier labs pushed the idea of guard rails and harnesses as the solution. It sort of works, most of the time, and it lets users yolo all their secret information and get things done fast.