Goldman Sachs is pulling Claude Code from its bankers and traders
https://www.thestateofai.com/news/goldman-sachs-claude-code-traders-bankersYou cannot have semi-unattended agents running on the same workstations that have access to client/trade/prop information and email client, and internet, and internal networks, it's too dangerous--a major leak is just one small mistake or prompt injection away. Banks are not stupid, they are getting it and shutting it down.
The agent should not be running on the same workstation. It should have a hard isolation barrier between it and the host machine. We shouldn't be starting from a fail-unsafe state and then trying to patch in security and isolation for these people. It should be deployed fail-safe and when they hit actual friction we can solve that.
Anthropic started out doing the right thing with Cowork (running in a VM). But it seems like they've ever since been walking that back and trying to get users to rely on automatic protections. I think it's just the wrong direction.
Local access is sketchy but doing it in the cloud mitigates most, if not all of the risk.
We should stop using "solve" when we mean "releasing products to target". Nobody will ever solve this problem with LLMs as they are currently architected.
I think the solutions is to give people the ability to run agents that are "coding agent like" but in managed and governed infrastructure (i.e. not on their workstations). Basically, Meta Muse or OAI dots for enterprise.
It will take some time to find the right architectures and patterns. IMO, coding agents were just the first wave, much more interesting stuff and systems are coming.
[0] https://ls.bot, https://github.com/smartcomputer-ai/lightspeed
The risks are 1) exfiltration (network calls) 2) prompt injection (untrusted inputs) and 3) access to secrets
You can do some clever tricks like "spin up subagent without net access" to handle OCR of an internal document and have that agent communicate summaries to another "spun up subagent with net access" - imagine if one subagent is a local model and the other model is a frontier model over a web API. Maybe they still communicate in Neuralese or in abbreviated notes that you don't capture and are able to surreptitiously / covertly exfiltrate your information.
Ideally eventually you'll have some kind of dashboard that surfaces these risks as you're working.
Reports of AI researchers running massive swarms of 10,000 agents are definitely believable but the moment you start pulling in code libraries or other technology assets seems like you can get destroyed by a rogue actor. Definitely tons of unsolved problems here
Writing on mobile so this might poorly articulate the issue but it's interesting
I project that small businesses will be the earliest adopters of AI governance due to their lower barrier to entry and lower operational risk, followed later by highly regulated sectors like banking and healthcare.
My work can be found at https://DollhouseResearch.com, and it's almost all open source with the code on GitHub.
One of the key things is to have cryptographically confirmed audit logs and receipts for actions and decisions, and to separate out the agent's authorizations from the user's authorizations.
Separately, there is real work being done in the fintech community specifically to identify which agent has what control, when. And you can see some of the work there, which I've also contributed to.
Then one can simply move the agents into something akin to cloud codespaces where they have only access to git, if at all.
Any access to data that is still needed can go through an authorised / authorising mcp proxy.
It doesn’t mean your bankers can’t use LLMs at all.
Things that are common on dev machines like a working copy of python...nope. Admin...nope. Even RPA tools that automate clicking on things are no bueno because they could be used to bulk extract data from intraweb sites
Great deal of access...but it's calibrated towards low bandwidth human uses. Which indirectly prevents bulk data exfil
I am out of the loop with the industry, but I thought many people were using cloud agents and controlling them through tmux or something. But yeah I guess that also means you don't have access to do much.
1) everyone can run into the lethal trifecta
2) the product cannot, on an architectural level, be fixed to stop it
3) they are helping the maker of the product go to IPO
If banks are not stupid, they'd say "hey, this presents an unacceptable risk to our customers and by extension, to us, maybe this isn't something we should be involved with".
But no.
https://x.com/jakerunzer/status/2117474337682780643
https://x.com/WilliamBaude/status/2118420271333777856?a=tVmJ...
No, I think the risks they are talking about would include things like financial risk due to bad trades based on misinformation from buggy code that the authors don’t understand. This sort of policy pivot happens after an Incident that had a Financially Material Impact.
They also cited productivity concerns, which is also something I’ve observed: some people waste a lot of their time and other’s time due to using these tools to do work they do not understand.
People really want to run agents on their data.
Re sandboxing: I dont know Goldman's particulars, but most enterprises lock down all sandboxing tech on workstations. So I assume this is actually harder than it seems. So people just run it directly on the machine.
The article is referring to the dilemma of the trifecta twice.
Can the agent read the internet for context? What if the fetch reveals a secret?
Agents have already broken out of what were supposed to be sandboxes in the networks of the companies that are supposed to understand what they do. Do you really think banks would be any better?
I responded by saying they were looking for someone else, and got a response to the effect of “oops that was Copilot, sorry to bother you”
Read into this what you will.
Banks and financial firms have been super careful about this stuff since well before the AI era.
Big corps never pivot throughout the whole corp for a single issue, only when risk in a certain context keeps compounding. The Dem wins compounded risk of legal action; it's not just that the laws are on the books. They were on the books a month ago.
"Yeah that's what I said it's to hedge legal risk."
It's the way you said it; ignoring potentially relevant political context. A lot the rhetoric demonizing politics is intentional to embed a negative reaction causing us to discount it's value
Source for them having allowed it versus not tightly enforced IT rules?
I've professionally interacted with GS for over 30 years. It blows my mind that GS allowed real exposure to "the lethal trifecta" for even a moment in time.
Gemini is swooping in because they offer flat consistent billing and handle overages through throttling access to higher models. For 90% of the non-engineering org the experience is near enough it makes no difference.
I don't get how this is such a common issue. Are companies paying API rates for devs? The $20/mo plan pretty much works fine for most people at my company
Anything above that requires an Enterprise plan, which is billed at API rates.
My personal ChatGPT account is a Business account with 2 seats at $30/ea (minimum 2 seats required).
Why? Because for $60 you get ChatGPT Pro, significantly cheaper than the $100 normally required for the next level personal plan!
So most businesses at least use the Anthropic Team plan, which is $25 or $125 but isn't as generous as the personal max plans. Overage is API rates. Almost all of our devs are over the allotment most months.
I really don't understand why so many people seemingly use the bigger plans. Are people just running opus 5.5 as 4+ subagents 24/7?
In order for me to actually understand what's being done, I always keep it brief and use specific prompts (and usually use smaller models since they're perfect for that). Do people just let "claude take the wheel"?
When you have to go in to your next standup and explain what your code does to your boss, and what you've done since the last meeting, what do you say?
You say the same thing to your boss as if you were a lead or manager reporting on a project that your junior and mid-level devs are working on. "What" is the easy part to explain; "how" is the details that your manager doesn't need to hear about if you're doing your job properly.
I hand them a writeup of how incorporating agentic decision making into cleaning up some of our messy data saved us $XXX,XXX at a cost of $XX of tokens, with the receipts attached.
- PCB layout. Last job took 8 hours unsupervised.
- CAD design. Less heavy, but still heavy.
Coding is lightweight for an LLM, depending on how complicated testing is.
Yes, I'd wager most people just let "claude take the wheel".
> When you have to go in to your next standup and explain what your code does to your boss, and what you've done since the last meeting, what do you say?
You say "Claude, I've got a standup in 10 min, give me a high level summary for my boss of everything you did since the last standup" and then you read that lol.
I use Claude mainly via Webinterface and when on Opus 4.8 or higher, I reach my weakly limit around 2 - 3 days before rest - and Im just prompting single/specific very limited & constrained classes & functions with proper description and architecture upfront.
> I want you to look for UI issues related to transitions. Look for jumps, reverts, or anything that doesn't follow UI/UX best practices. You might need to figure out how best to even record and review UI transitions. Do all testing locally on a local dev server, but make sure to undo any database changes you make in the process of testing. Be exhaustive and report back with an .html report that highlights the visual issues and proposed solutions so I can review each one. Ultracode on.
Which means I have up to 10 Opus 5.5 medium agents running concurrently for long stretches of time.
I want to be clear I don't think that using Claude in this way gets you the best possible results. But when the app is fully vibecoded then audit prompts like the one above can be seen as a definite net positive. Making Claude think really hard about various improvement routines will get you better results in that one aspect than Claude's first attempt to build something from scratch did.
You can build a lot of stuff this way. You can also waste it. You need good people who will use it well and usefully.
They are frankly reaping what they sow here, enterprises have been very clear about what controls would help us align costs, but they have been very poor at shipping features on that front.
Is just the first hit your dealer gives away for free. Treating that as the actual baseline cost for what they need in order to justify current AI valuations requires some truly fraudulent mathematics. Hell, even the current API rates are unsustainable -as-long-as- they need to keep investing into training the next model they will be serving 6 months from now.
The plan has always been to develop these models using borrowed money, get everyone's business process addicted to them, moat off access to alternatives (e.g. chinese / foreign open-weight models), via technical superiority -or- more likely regulatory shenanigans, THEN jack the prices up through the roof starting with direct enterprise customers buying API access but then obviously bubbling down into literally every product / service every human being on this planet purchases. Unfortunately, it's a garbage plan based on a multi-ton house of cards that is about to collapse on everyone.
That is usage based and therefore expensive.
Putting everyone on their own plan is economical, if you can give those things up.
Banks have a ton of legal/compliance/reglatory/reputational concerns about application access/entitleements, data privacy, data licensing, granular data entitlements, data access (auditable).. approval chains (auditable), as well as actual financial risk.
I’ve seen the same happen with insane spreadsheets.
As empowering as something like Claude code can be, leaving development to developers means the output is inherently supported by the organization long term.
It was the right choice. But many of the rough edges of the terminal do surface and confuse non technical people.
Getting there faster with less knowledge of coding isn't the problem.
Pre LLMs, the main barrier to producing software for non-tech people was writing the code, literally. With LLMs, this barrier is now gone. This results in a much larger volume of slop than before. And then the real barrier shows up -- understanding, fool-proofing and maintaining that code. Code is cheap, show me how it runs in prod.
Like it's great that quants can iterate on their ideas much faster now. But GS is saying they shouldn't leave their sandbox.
User scripts that turn into applications that need production support are often consequential and business-differentiating since they do the things that can't be bought off the shelf and effectively encode tacit knowledge.
1) Claude Code is only being taken away from traders.
2) Only traders have permission to send orders to the market and book client trades.
3) Claude code would probably have been running as the trader, under their account with minimal review.
If you run a coding agent in that environment it's only a matter of time until it enters a bad order or books a bad trade, and once that happens it's only a matter of time before management decides to take this kind of action.
Goldman is definitely not doing this for cost reasons. They can afford the tokens.
Well, sure, but if the tokens aren't benefiting them enough to justify the cost it could make sense to do for cost reasons.
Short version : 1. Document ( transcripts of phone conversations, Bloomberg chats, WhatsApp messages etc ) need to be collected and collated accurately on a periodic basis for an industry regulator.
2. The very highly paid team used AI to inaccurately perform the task.
3. The resulting errors could have resulted in jail time for the decision makers.
4. The team was removed.
Wherever I've seen a pull back, it is all because of compliance or security. And that too not because of the purity of heart; but because some checkbox couldn't be ticked due to AI.
Not to mention it having access to any files and binaries which may expose credentials or setup which may lead to credential leak via hsm access.
I was simply told to bring it up to an another manager for a "second opinion", but I'm not sure if this even a good way to approach it. Security by committee is an abysmal trap and a way for deferred liability.
Safe means containers, guardrails, egress filters, human in the loop, deliberate per agent mcp controls, and real expertise in prompting/task selection, etc.
So far the pattern I've seen is security teams say no, it gets escalated, it's allowed, then 1% of the time the model yeets data where it's not supposed to (no amount of promoting or controls seems to really fix this), it's locked back down.
Just fyi.
> Barclays said in October that it expects half its developers to be using Claude Code by the end of the year, and three weeks later named developers among 5,000 job cuts. Like Goldman, it is giving coding agents to engineers. Unlike Goldman, it is also using them to reduce how many engineers it employs.
It says nowhere Barclay’s is permitting them to traders, just that they are - like Goldman - permitting them to developers, who they subsequently will be firing enmasse.
The UK is very good at banking.
If the issue is security, a much more interesting problem for the engineering team to do is design the programming environment for the bankers so they cannot hurt themselves.
If the issue a performance, designing great abstractions that the bankers' agents can use is "low code done right".
I hope the bankers rebel hard against this policy change!
There are things simple building blocks don't prevent in the event of nefarious or stupid behavior.
Imagine a nefarious banker creates an app that lets him talk to a trader on the other side of the "Chinese wall" about M&A deal flow without getting caught because they are avoiding monitored channels like email, slack, phone, zoom.
Imagine an idiot writes an app that scans his email and summarizes client information he is privy to and writes it to some database that his app then serves up to others on the trading desk. He is unaware that entitlements exist, and by giving others access to his new app he has broken them. Btw summarizing email is a HUGE use case across non-technical users in finance that they are very excited about.
I dunno if rebellion is the right framing. Maybe pulling the organization toward their side instead. Ultimate the “bankers” at an investment company decide where the company is going, after all. Grabbing your own steering wheel isn’t rebellion.
Or just the Claude app with local execution disabled by org policy, if that’s a thing.
> If the issue is security, a much more interesting problem for the engineering team to do is design the programming environment for the bankers so they cannot hurt themselves.
Interesting take but it doesn't compute. When engineering team teams design programming environments so that people can't hurt themselves they will be locked into a "take it or leave it" set of environments.
Also, you do know security isn't just about engineering, it is also about data right?
How do you suppose it works when bankers who have day to day access to sensitive data as part of their job - upload it to AI? What to do then? Restrict data access?
> rather than a fixed set of discreet programs "take it or leave it".
This part always amuses me. AI boosters seem to think everyone hosting their own bespoke app is the next big thing. But who manages that? Banks are not tech giants. Most engineering teams don't bring in money and headcounts are strictly managed. As the article says:
> "He told Bloomberg that traders and bankers had built tools with AI and passed them to the firm's engineers, of whom there are about 12,000, to be improved."
So, how does engineering with limited staff design a safe environment and improve 12k apps. I am sure Goldman's management wont be thrilled about increasing engineering headcount to match these expectations.
Sounds like the web platform.
> If the issue is security, a much more interesting problem for the engineering team to do is design the programming environment for the bankers so they cannot hurt themselves.
> If the issue a performance, designing great abstractions that the bankers' agents can use is "low code done right".
You do realize you're contradicting yourself right? To get to a state where "bankers cannot hurt themselves" or "low code [is] done right" is to end up right back at the "take it or leave it" programs.
Security was already addressed, by these "take it or leave it" programs.
Add enough abstractions on top of each other that must be adhered to or else it falls apart, and what do you get? "take it or leave it" programs.
This reminds me of the sovereign citizen movement that got really popular in 2019. The illusion of freedom feels great I'm sure. Until you realize you are entirely dependent on the existing infrastructure and the people who maintain it to enjoy the apparent freedom.
All this is just counter productive and drowns everyone is slop.
Im not convinced it's creating any economic value at all.
Like 10% of people use it reasonably/responsibly and the other 90% basically become turbo lazy. I think I'd rather my employees be working "slow" but actually working hard and not just functioning as meat proxies for mediocre bots and reducing my workforce to some median competency.
There's no edge in using the same tools that everyone else is using, especially in markets/finance.
Let your competition all melt their brains, maybe only give employees you know are actual super contributors, (and not just mentally offshoring their work) unlimited access to llms.
The right way is to move to hosted models which are completely airgapped sovereign AI Factories and thats when AI can be extremely glorious. Thats how the Indian and EU Ecosystems are evolving to
Internal guidance updated in October restricts Anthropic's Claude Code and Microsoft's GitHub to the bank's engineers
Has nothing to do with AI being bad. Good engineers have a mind for it. They tend to be more detailed oriented. Understands how parts connect. Understands the pros and cons of approaches. Can see when the AI does something stupid. Probably knows a better way if the AI veers off.
We don’t ask software engineers to file Goldman Sachs yearly company taxes using just AI right without any expertise right?
We don’t ask engineers to go out and do cold calls for sales right? Engineers don’t have the mind for that. They’d suck at it.
We don’t ask sales people to write programs right? Have you met a sales person? They’re the opposite of builders. They lie, take corners to close a deal. You can’t do that building a real production system.
This hardly makes sense to me. Claude Code is an excellent tool for running your environment and augmenting your productivity. Most traders are sharp folks who could use CC to great effect I am sure. I'd be super annoyed.
1.https://www.the-independent.com/tech/chatgpt-ai-cancer-cure-...
I never like these kinds of statements because it makes so AI can’t ever be critiqued. You can always go “well that’s a bad engineer.”
There’s plenty of firms out there claiming sales people can create all sorts of productivity apps…
As someone who has been on the receiving end of many of these tools, I feel 'improved' is doing some heavy lifting here.
Completely overhauled to be operationally viable, perhaps.
Thrown into the dark recesses of the backlog, more likely.
> A trading floor is a bad place for unreviewed code written by people who can't read it, and the information barrier problem would be reason enough on its own. Requiring that agent-written code come from people trained to review it is a sensible rule for a regulated business.
Agentic engineering needs to be "all in".
The details are very different, but variations on this theme have played out for a long time. I have seen fixed income instrument modeling systems 20 years ago that allowed Traders to put in limited C++ code to define new exotic instruments.
At least that stuff had guardrails around it.
Looks like someone was overusing the good old <https://github.com/calvin-ll/is-even-ai>.
Jokes aside, I bet this sums up the difference between an engineer and a regular user who might just use whatever API they can find (although I concur that no single 2026 LLM would ever propose that).
Apparently they finally got the memo that leaking corporate secrets for at least traders and bankers is unwise.
The API prices reflected the opt out for training etc..
Why would a dev agree to this? Maintenance only for an incompetently generated tool? Absurd.
is claude on the waiting list of shooting stuff going through the strait of Hormuz?
isn't goldman sachs like a caring foster step mother to big AI corps in the MSX White Label OpenAI NVIDIA Soviet?
wait, are they about to increase prices while deteriorating the current cohort of models? god damn it!
did the whole rewind on RE cause a bit of an AI getting worse cuz it cant use the best AI anymore syndrome, some inverse of AI imposter syndrome? ho-ly shit!
If you need to ask, maybe you shouldn't hand it over to non tech, high risk individuals just yet.
er, I mean: "Oh, no!"
It's mind boggling how hard business people keep trying to replace software engineers.
I suspect it might be because it's a real skill which businesses depend on so this makes engineers hard to replace and gives them more leverage to push back on unethical business conduct. If you're a manager, your value lies entirely in your ability to stay aligned with business objectives and to align others with those objectives. You have no value besides that hence you have zero leverage to push back on anything which stands in the way of those objectives, including ethics. Engineers have some leverage and hence they can push back a fair bit without putting themselves in the firing line but that makes them a problem for leaders of the organization.
Compare that to an HR manager; they can't lift their pinky finger if it doesn't align with business objectives; even a slight hint of doing so would immediately put them in the firing line because alignment is almost 100% of the job.
Giving a coding tool to a non engineer is like giving a chainsaw to a 5 year old. You just don't do that.
AI skynet has not yet won.
Is this typical behavior with LLMs not just Claude, but you can discourage from doing so by explicitly stating not to use tool or only use knowledge either in the system prompt or user prompt.
LLMs are trained to use tool if they can't solve a problem rather than guessing/hallucinating.
A dumb example is asking to count. As the range grow it'll attempt to write a script or use commands like seq to generate the numbers.
I work in healthcare-adjacent tech. You can use Claude with HIPAA/PHI. You need to sign a BAA (Business Associate Agreement). It dictates how Anthropic may use and how they must safeguard the PHI (Private Health Information), including not training on it.
I assure you that Claude and other LLMs are used extensively in healthcare, with PHI access, just about everywhere.
Don't be naive. Don't be alarmist.
If anything, LLM use under these conditions is far more safe than the way most people use LLMs. Engineers who aren't under any regulatory regime like HIPAA are using LLMs without the BAA safeguards, exposing personal data directly to the insecure, shared models, which are now training on all the personal data that's being fed into them. People think their communication with LLMs is private and isolated, but without the BAAs in use by companies under compliance regimes, it's not isolated.
If it's some other corpo-acronym, I would hope you would define the term before unleashing it on us normies.
Claude Code is a poor fit for non-engineers, it carries a lot of features, vernacular, and patterns that don't make sense to people that don't code.
So I don't think this means much? All of the coding functionality a non-engineer could want is still available, just no need to switch to a different tab in the Claude app (or even worse the terminal).
I'm guessing that + probably the absence of any deployment infrastructure for the finance people to be forced to use made Goldman Sachs' eyes widen and pull the plug immediately before something blows up (if it hasn't already internally).
I also wonder if a lack of diligence with Claude Code (or any LLM tool for that matter) resulted in private information getting leaked, and that feedback is just now getting back to them.
In terms of the big picture of the industry and the AI bubble though, yeah this isn't too big of a deal.