Hacker News new | past | comments | ask | show | jobs | submit
I get the impression that every AI lab is desperately trying to figure out how to unambiguously separate instructions from data in their token streams. The fact that they haven't managed to yet suggests to me that it's a very, very difficult problem.
I think what's interesting here is that they've shipped the product despite these glaring security flaws. I've noticed that in my own professional life, at some point after the pandemic people stopped caring about security as much. Issues that would have (and should have) blocked a product launch were swept under the rug.

I suspect this comes with the territory of enshittification. As an industry we're trying to wring every last dollar from every last eyeball and we've discovered that building secure systems doesn't actually move the needle very much.

> I get the impression that every AI lab is desperately trying...

Of course.

I wonder how we managed way back in the day to produce systems that can handle untrusted inputs and reliably instruct a dumb-as-bricks CPU what to do based on those inputs. Must have been black magic lost to the mists of time.

>reliably instruct a dumb-as-bricks CPU

Yeah...a "dumb as bricks CPU", which is obviously something frontier llms are demonstrably not. Like, you're not making any sense here. None of the things that make this possible with CPUs is remotely relevant here, and the fact that you don't seem to understand this but act so smug is strange.

loading story #49224015
If you can figure out how to separate instructions from data in LLMs you should ship the first agent system that's guaranteed protected against prompt injection. You'll make millions.
Why not just have distinct input streams, or a metadata stream which annotates text in the main stream according to priority in case of conflicting instructions?
loading story #49227172
It’s pretty simple. Both the intake and the output of the LLMs are data and they shouldn’t drive an actuator system (their output shouldn’t be instruction). We already have the same structure in organizations where there’s an army of analysts for information gathering and processing and then the executive department tasked with decisions.

We have even observed that the most effective LLM usage is when paired with an expert in charge of the goals. Dark factory and other automated harnesses (specs engineering and what not) seem to be a dead end. The most impactful approach to this date is an interactive conversation as a succession of small and verifiable tasks.

loading story #49223827
Yeah, this matches what I've learned over the past couple of years from reading some of your blog posts and reading your interactions in comment threads here and elsewhere. You're a politician, rather than a truthseeker.

The absolute most I've seen from you in response to an extensive teardown of your argument, supporting evidence, and subsequent conversational judo was a «Wow. That was well phrased.» and no subsequent change in your publicly-expressed opinions.

I'd do more than gesture at the relevant lesson taught to us by Google Fiber, Tesla, SpaceX, etc., but you'd not be publicly moved, so it's a waste of time.

> You're a politician, rather than a truthseeker.

Justify that.

Also, which "extensive teardown" are you talking about there?