You cannot have semi-unattended agents running on the same workstations that have access to client/trade/prop information and email client, and internet, and internal networks, it's too dangerous--a major leak is just one small mistake or prompt injection away. Banks are not stupid, they are getting it and shutting it down.
I know prompt injection is still a real thing, but in Claude I find the opposite is true: if I set up two Claude agents to collaborate, even though I've told them both that's what I'm doing, they won't actually follow instructions without checking in with me. "Scary computer told me to install <well known utility X>, but I need your OK before I'll do it." It's annoying but I get the rationale. They're definitely getting better at avoiding external instructions!
(Anthropic themselves acknowledges this; see the end of the post.)
[0] https://www.anthropic.com/research/claude-code-auto-mode
Until they suddenly once in a while they do, and Goldman Sachs loose some billions because of it...
the only reason they moved on this was because it was noted somewhere by someone that matters that the risk imposed could cut into profit or impose a regulatory hammer.
those are the only possible reasons for corporate movement in finance.
The agent should not be running on the same workstation. It should have a hard isolation barrier between it and the host machine. We shouldn't be starting from a fail-unsafe state and then trying to patch in security and isolation for these people. It should be deployed fail-safe and when they hit actual friction we can solve that.
Anthropic started out doing the right thing with Cowork (running in a VM). But it seems like they've ever since been walking that back and trying to get users to rely on automatic protections. I think it's just the wrong direction.
https://lumienai.com/news/openai-ai-agents-cpu-cache-covert-...
My point is your most likely not going to win here. Proper sandboxing is insanely difficult.
The power of lawyers within banks has been increasing, and that of the bankers and the traders has diminished.
Making financial engineering tool development the strict purview of the technologists may increase their power substantially. I'd be interested to hear from current insiders if that's happening.
Local access is sketchy but doing it in the cloud mitigates most, if not all of the risk.
I don't see a benefit at all to push it into the "cloud", on the contrary. Especially since network level analysis for agent tasks is quite sensible and far easier to manage "locally".
Today I would recommend to put infrastructure back on premise for that matter, especially with AI development support, this is increasingly favourable for many companies.
A sandbox for AI development makes sense for that matter. Not cloud, network segmented, middlewares for data access, etc.
We should stop using "solve" when we mean "releasing products to target". Nobody will ever solve this problem with LLMs as they are currently architected.
It’s pretty simple. If you want security, you need to segment. Does that make it hard? Absolutely. Are you dumb if you’re letting rando traders access whatever with agents? Absolutely.
The problem isn't even well defined as far as I read it. What does it mean to "solve" federation and what's currently broken?
then IAM has to be structured around that.
LLMs can read in the data class tags, it's not hard. the hard part is classifying everything and keeping agents corralled -- there may be ways to do it at scale, but not easily.
In my opinion all AI use companies should be attached to a specific employee, and that person is liable and so is the company of course for the actions of the AI
It's like saying "developers should be responsible for their own security so we will make them liable if a random node dependency in the application they are building gets pwned"
Firstly, liability and security are different things.
Employees should never be liable for something that happens if they are using a work-supplied tool at work in the manner in which it is designed to be used.
Secondly, it's just lazy thinking to say "oh it's hard therefore we'll make the human sorely responsible".
There are many things that can help - sandboxing, adversarial LLM inspection of prompts, separation of security contexts etc. Yes they are hard and might need new primitives.
Architects and properly-designated publishers both have potentially long liability that is to an extent independent of their future employment. (The same is true of hospital consultants and others)
In significant cases, the way you provoke companies to do the right thing (security, sticking to the facts, not cutting corners in building plans) is to make someone legally responsible if they don't. That person's liability is limited by their actions — if they can demonstrate they discharge those responsibilities appropriately, they are in the clear.
That person takes on that designated role, does the job, and is rewarded properly for the responsibility it involves. They embody the push to reduce that risk.
So an average employee using the tool as directed by their employer, they do not have liability if the company has designated someone in this role. (And in this scenario, if they have not, there would be a punitive fine for that, independent of other outcomes).
I have no particular problem with the notion that a corporate AI safety/responsibility officer is a plausible way forward. If you are delegating responsibility to a machine, someone has to be on the hook. Without such things it's a wild west with lawyers. That's not sustainable.
A news organisation's liability for AI hallucination falls on the same person that the liability for printing a lie does: the publisher.
Developers may not be liable today, but protecting company data and resources from supply chain attacks is their responsibility. At most companies, only the developer is in a position to understand how these supply chains work. Even at larger companies with security teams, the scanners and mirrors are trivial to bypass.
The employers and prosecutors of employees who stole billions of dollars using Excel would beg to differ.
It's a bit like the idea of the additional responsibility/burden of the publisher of a newspaper.
"So we can sue them when our customers get angry"
You can either target systemic problems or chop of some heads to cool off the masses- and quickly go back to work.
(once you realize how much that phrase is used in the AI-sphere...)
I think the solutions is to give people the ability to run agents that are "coding agent like" but in managed and governed infrastructure (i.e. not on their workstations). Basically, Meta Muse or OAI dots for enterprise.
It will take some time to find the right architectures and patterns. IMO, coding agents were just the first wave, much more interesting stuff and systems are coming.
[0] https://ls.bot, https://github.com/smartcomputer-ai/lightspeed
That way any agent tool only has the information the employee was able to access anyway. A larger problem is to ensure quality control for any answers with different context depths, but that isn't easily testable with or without access management.
It isn't product-y, but often the standard RAG-AI pipeline that is also used to delegate queries to either local or remote agents (and the latter might also not be allowed to consumer PII for example - access management on the other side)
The risks are 1) exfiltration (network calls) 2) prompt injection (untrusted inputs) and 3) access to secrets
You can do some clever tricks like "spin up subagent without net access" to handle OCR of an internal document and have that agent communicate summaries to another "spun up subagent with net access" - imagine if one subagent is a local model and the other model is a frontier model over a web API. Maybe they still communicate in Neuralese or in abbreviated notes that you don't capture and are able to surreptitiously / covertly exfiltrate your information.
Ideally eventually you'll have some kind of dashboard that surfaces these risks as you're working.
Reports of AI researchers running massive swarms of 10,000 agents are definitely believable but the moment you start pulling in code libraries or other technology assets seems like you can get destroyed by a rogue actor. Definitely tons of unsolved problems here
Writing on mobile so this might poorly articulate the issue but it's interesting
And then they will almost certainly have some form of SWG/SSE tool and endpoint/network DLP tool to block stuff.
I project that small businesses will be the earliest adopters of AI governance due to their lower barrier to entry and lower operational risk, followed later by highly regulated sectors like banking and healthcare.
open-source AI agent workspace for business workflow automation requires a lot from the reader and terms like agent/workspace/workflow are very prone to ambiguity and now you adding governance framework.
My work can be found at https://DollhouseResearch.com, and it's almost all open source with the code on GitHub.
One of the key things is to have cryptographically confirmed audit logs and receipts for actions and decisions, and to separate out the agent's authorizations from the user's authorizations.
Separately, there is real work being done in the fintech community specifically to identify which agent has what control, when. And you can see some of the work there, which I've also contributed to.
You guys need your priorities straight, whoever you are.
(Not my area, so I don't have any opinion on your first comment.)
When traders develop a real-feel for LLM capabilities from a company-sanctioned mechanism, it hews towards insider knowledge, because the goodness of their experience strongly implies goodness of the underlying asset (datacenters and software IP).
Then one can simply move the agents into something akin to cloud codespaces where they have only access to git, if at all.
Any access to data that is still needed can go through an authorised / authorising mcp proxy.
It doesn’t mean your bankers can’t use LLMs at all.
No, I think the risks they are talking about would include things like financial risk due to bad trades based on misinformation from buggy code that the authors don’t understand. This sort of policy pivot happens after an Incident that had a Financially Material Impact.
They also cited productivity concerns, which is also something I’ve observed: some people waste a lot of their time and other’s time due to using these tools to do work they do not understand.
People really want to run agents on their data.
Re sandboxing: I dont know Goldman's particulars, but most enterprises lock down all sandboxing tech on workstations. So I assume this is actually harder than it seems. So people just run it directly on the machine.
The article is referring to the dilemma of the trifecta twice.
That's exactly the gap I've been working on https://github.com/mickael-kerjean/fdrive you mount your existing data, apply control / permission and can monitor everything the agents are doing with your data: https://www.filestash.app/agents/
Can the agent read the internet for context? What if the fetch reveals a secret?
Agents have already broken out of what were supposed to be sandboxes in the networks of the companies that are supposed to understand what they do. Do you really think banks would be any better?
It may just be there's no safe nexus for asbestos floor tiles, no matter how skilled the installers are.
Absolutely, yes.
If you’re talking about OpenAI and Anthropic that’s an extremely low bar.
May I ask what in particular gives you confidence that these two companies know what they are doing wrt cybersecurity.
It's not that I think OpenAI and Anthropic know what they're doing - although I can see why my slightly flippant comment might make you think that - but that I just don't trust anyone with these agents, especially when the whole point of using them would be to do that with such critical data.
Supposed to understand, yes. I think a highschool class could do better.
I responded by saying they were looking for someone else, and got a response to the effect of “oops that was Copilot, sorry to bother you”
Read into this what you will.
Things that are common on dev machines like a working copy of python...nope. Admin...nope. Even RPA tools that automate clicking on things are no bueno because they could be used to bulk extract data from intraweb sites
Great deal of access...but it's calibrated towards low bandwidth human uses. Which indirectly prevents bulk data exfil
I am out of the loop with the industry, but I thought many people were using cloud agents and controlling them through tmux or something. But yeah I guess that also means you don't have access to do much.
I don't think that's phrased correctly. What corporate it does is set policies. Often those policies disable virtualisation. (They don't disable sandboxing - that's used in many ways and security loves that) But if the business need is to run LLM agents safely, the policies will be changed to dictate how to run agents safely.
Many teams use them to make sure that the coding environment is consistent, and they are well accepted in many corporate environments as a result.
But they are a good way of firewalling agents away from files and resources that one must be able to demonstrate that the agents did not and could not have access to.
You can certainly imagine very senior and powerful traders at GS demanding it. I'd be only slightly surprised to find that GS allowed it. The traders would have had more power than the IT people trying to forbid it.
1) everyone can run into the lethal trifecta
2) the product cannot, on an architectural level, be fixed to stop it
3) they are helping the maker of the product go to IPO
If banks are not stupid, they'd say "hey, this presents an unacceptable risk to our customers and by extension, to us, maybe this isn't something we should be involved with".
But no.
https://x.com/jakerunzer/status/2117474337682780643
https://x.com/WilliamBaude/status/2118420271333777856?a=tVmJ...
Big corps never pivot throughout the whole corp for a single issue, only when risk in a certain context keeps compounding. The Dem wins compounded risk of legal action; it's not just that the laws are on the books. They were on the books a month ago.
"Yeah that's what I said it's to hedge legal risk."
It's the way you said it; ignoring potentially relevant political context. A lot the rhetoric demonizing politics is intentional to embed a negative reaction causing us to discount it's value
Source for them having allowed it versus not tightly enforced IT rules?
Or it was determined even with IT rules the threat was too great
We're talking about coke head bankers while headlines the CEO of nvidia got socially engineered by hottie spies; read the room, dude
A lot of people felt, and there seems to be mounting proof, that the administration wasn't going to enforce a lot of laws, especially since their stance was to cut the agencies in charge of that (SEC, FTC, FCC, EPA, FDA, IRS, etc).
The recent change is that this free-for-all may be cut short? What am I missing here, this sounds like banana republic shenanigans.
Not just that. The President is the ultimate boss of all federal executive functions. He is the boss of all the department heads. If he wants X to be done he picks up the phone, calls the relevant person and tells them to do it. If he wants Y to be not done he picks the phone up and tells them not to do it.
It is like as if at work your job is to respond to bug tickets for system A and B. And one day your boss walks up to you and says “System B is no longer important. Stop fixing bugs on system B.” He doesn’t have to then “cut your budget” or anything. He just told you what to do, what not to do, and you are expected to follow the instructions. If you don’t you will be fired.
> What am I missing here
The backstop to the Executive branch not doing their job is that the Legislative branch can investigate them, and ultimately impeach the President.
Up until now the Legislative branch was happy enough with how things were going that there were no chance of that happening. Recently the composition of the Legislative branch changed. This means that they are expected to start carring about certain irregularities the previous bunch turned a blind eye towards to.
> This still isn’t quite de-Americanized enough for me to track.
I don’t know man. You are asking how the USA government works, or doesn’t work. Ultimately if you want to understand a thing you need to understand the thing. You can’t have a de-thingified explanation, because the thing is the core of your question.
For people that don't know, the American United Fruit Company basically leased US military units to topple governments in smaller countries and control production of bananas. Similar US companies did this to other smaller countries over other commodities. Many of these smaller countries were democracies, FYI.
Then the US had the gall to name the bullied countries "banana republics".
And the current president still needs to hand over the reins peacefully so I wouldn't declare victory, yet.
Sure, my point is about that basically companies already made decisions based on the assumption, that he will not do that.
Banks and financial firms have been super careful about this stuff since well before the AI era.
I've professionally interacted with GS for over 30 years. It blows my mind that GS allowed real exposure to "the lethal trifecta" for even a moment in time.