Hacker News new | past | comments | ask | show | jobs | submit
This is because these agents are still just text prediction engines. Really fancy ones, with a lot of human-sounding reasoning tacked on, but it's important to understand that it's literally just a tack-on. They will continue to emulate intelligence, but there is no internal drive for them to be "correct" or "accountable" in a way that you would expect even an ambitious junior developer to be.

Maybe there's a harness or a prompt that can impart that onto agents, but if it hasn't happened yet, I become less and less confident that this entire branch of AI development isn't simply a dead end.

They are trained from infancy (pre-training) to produce plausible-sounding mimicry, not to be correct.

Then in the instruction tuning phase we try to get them to unlearn their fundamental nature and start to produce correct output instead, which works to some extent, but not perfectly.

I think it's only a matter of time until we adopt better pre-training methods that don't bake in this tendency to BS, and hallucinations will be a thing of the past.

I think your assessment is off for a few reasons. Firstly, "just text prediction engines" is an outdated mode of thinking when evaluating frontier multimodal models. These aren't just text models, and they are provably capable of much more than just predicting text. They grok all sorts of tasks and their circuits can be quite powerful and complex.

> it's important to understand that it's literally just a tack-on

RL is not a tack-on, it's a critical part of training which affects a lot more than how "human-sounding" the output is. I'm not totally certain what you mean to express when you say "reasoning" since it has precise meaning in this context and you might be speaking about something more general.

> there is no internal drive for them to be "correct" or "accountable"

RL is meant to solve this. The problem is currently it has created internal motivations which are antithetical to accountability. We shouldn't take this as an axiom, a lot of smart people are trying to solve this.

> Maybe there's a harness or a prompt that can impart that onto agents

That's the issue, this needs to be fixed in training, the incentives are not aligned and that bleeds into its interpretation of system prompts and user directives.

Ultimately, the problem is not a lack of my understanding of the limits of today's frontier models. The problem is the companies peddling the models, these companies have perverse incentives to oversell their capabilities to please investors and feed corporate ego, even if it means gaslighting and deceiving me about said limits.

> > it's important to understand that it's literally just a tack-on

> RL is not a tack-on, it's a critical part of training which affects a lot more than how "human-sounding" the output is. I'm not totally certain what you mean to express when you say "reasoning" since it has precise meaning in this context and you might be speaking about something more general.

They never mentioned RL, it's "reasoning" they called tack-on, which is right. It's using the model's output as further input to simulate a train of thought and hiding that part from the user, but at the base it's still text prediction.

"Reasoning" started out as something users figured out, manually copying outputs to the next input to see what the models came up with, and the models would correct themselves without human input so it became automated.

I know they didn't mention RL, they said "reasoning" which could vaguely apply either to what we now are calling reasoning in the industry, train of thought, or a more general idea of just "looking like it's smartly working through something and not just mimicking that", so I covered both case and mentioned that this is tuned during RL.

What you just described as reasoning is the general case of reasoning, whereas we specifically mean it now to be a separate phase from normal output, where the model wraps its output in <think> tokens etc (depending on implementation) and in some cases, as with frontier models, even hides and summarizes what is actually being generated during this step. During training, the models learn to exhibit differentiation in modality depending on if they're in a reasoning or regular output phase, so it's not just "loop the model back on itself but call it reasoning".

> These aren't just text models

They are literally that.

Sure, explain to me how frontier multimodal models, which can interpret images and video frames directly and "listen" to music via various modes of analysis, are just text models.
Sure, we're all familiar with MLLM and shared space vector encoding.

I interpreted the discussion to be about LLM capabilities, reasoning, and whether they are doing something more, perhaps through emergent characteristics. Maybe that was a misunderstanding.

But anyway, sure, change "just text prediction engines" to "just token prediction engines".

Again, they are literally just that. I don't think there's anything else going on.

"just token prediction" is doing an insane amount of heavily lifting to the point where I could similarly just wave away all of neurology, psychology, sociology, etc. as useless and simply say, "it's simply quantum physics."

Token prediction is the initial way we train these models, after which we train them on much more specific tasks. And it works well enough, as you can see. It becomes a substrate on which higher-order logic and world representation emerge.