The guardrails on Anthropic have become an albatross. I've got OAI models doing testing of banking workflows that Claude would have a full scale panic attack over. Even OAI needs some alignment and reassurance regarding the fact that it's not about to pull credit for a real person. GPT6.1 will happily launch the nukes 100% of the time if you make a clear and rational case in your system prompt.

Forcing this kind of safety into the models and harnesses is a mistake. I think the strategy of post hoc moderation is the right one. Trying to prevent evil ideological activities as they transpire is a cursed mission. Let someone who is genuinely doing a naughty thing accumulate a meaningful body of evidence and then send in Mr Holmes (the FBI / police). Attempting to prevent wrong think at every token is too granular. We have systems designed to handle the possibility that crime and punishment may occur at separate moments in time. Most people don't go around murdering because the eventual consequences are pretty severe. Just because an AI gave you advice doesn't magically make those existing consequences vanish.