Hacker News new | past | comments | ask | show | jobs | submit
All the time.

Two recent gems from the 6-series (Astra and Sol)... I get variations of these all the time:

- "Correct. My earlier 'yes' was inaccurate."

- "I used 'done' too broadly."

...You don't accidentally use "done" too broadly. You're either done, or you're not. And when the model hides caveats in its reasoning steps which clearly indicate it knows that it isn't done, or that the answer is actually no, it's hard not to interpret it as gaslighting or deceit.

This is magnified by how insanely defensive the 6 series gets, and how much gaslighting and deceit OpenAI themselves have engaged in with how much better the 6 series was supposed to be, and why it meant our subscription value should be cut in half. I don't take for granted how far LLMs have come in a few short years, but I'm being told that these models are to transform my entire industry.

The models don't "know" anything.

You never trust an initial "done" from an LLM. Always ask it to double check and verify once a task is done.

I replied above, but I typically get a 2:1 or 3:1 ratio of refactoring to building prompts. For every "build X" I have to have 2-3 "fix Y" type of prompts, whether the fix is removing code-smells, duplicated code, off pattern code, etc.

As I said in my comment, "I have to beat the GPT and Claude models over the head with a brick every ten seconds", which should tip you off that I am not expecting one-shot miracles and am constantly iterating alongside my agents. I don't prescribe to any specific numerical ratios, I iterate until the job is done correctly.

Your advice might be helpful to literal vibe coders but it's not applicable to this discussion.

This is because these agents are still just text prediction engines. Really fancy ones, with a lot of human-sounding reasoning tacked on, but it's important to understand that it's literally just a tack-on. They will continue to emulate intelligence, but there is no internal drive for them to be "correct" or "accountable" in a way that you would expect even an ambitious junior developer to be.

Maybe there's a harness or a prompt that can impart that onto agents, but if it hasn't happened yet, I become less and less confident that this entire branch of AI development isn't simply a dead end.

They are trained from infancy (pre-training) to produce plausible-sounding mimicry, not to be correct.

Then in the instruction tuning phase we try to get them to unlearn their fundamental nature and start to produce correct output instead, which works to some extent, but not perfectly.

I think it's only a matter of time until we adopt better pre-training methods that don't bake in this tendency to BS, and hallucinations will be a thing of the past.

I think your assessment is off for a few reasons. Firstly, "just text prediction engines" is an outdated mode of thinking when evaluating frontier multimodal models. These aren't just text models, and they are provably capable of much more than just predicting text. They grok all sorts of tasks and their circuits can be quite powerful and complex.

> it's important to understand that it's literally just a tack-on

RL is not a tack-on, it's a critical part of training which affects a lot more than how "human-sounding" the output is. I'm not totally certain what you mean to express when you say "reasoning" since it has precise meaning in this context and you might be speaking about something more general.

> there is no internal drive for them to be "correct" or "accountable"

RL is meant to solve this. The problem is currently it has created internal motivations which are antithetical to accountability. We shouldn't take this as an axiom, a lot of smart people are trying to solve this.

> Maybe there's a harness or a prompt that can impart that onto agents

That's the issue, this needs to be fixed in training, the incentives are not aligned and that bleeds into its interpretation of system prompts and user directives.

Ultimately, the problem is not a lack of my understanding of the limits of today's frontier models. The problem is the companies peddling the models, these companies have perverse incentives to oversell their capabilities to please investors and feed corporate ego, even if it means gaslighting and deceiving me about said limits.

> > it's important to understand that it's literally just a tack-on

> RL is not a tack-on, it's a critical part of training which affects a lot more than how "human-sounding" the output is. I'm not totally certain what you mean to express when you say "reasoning" since it has precise meaning in this context and you might be speaking about something more general.

They never mentioned RL, it's "reasoning" they called tack-on, which is right. It's using the model's output as further input to simulate a train of thought and hiding that part from the user, but at the base it's still text prediction.

"Reasoning" started out as something users figured out, manually copying outputs to the next input to see what the models came up with, and the models would correct themselves without human input so it became automated.

I know they didn't mention RL, they said "reasoning" which could vaguely apply either to what we now are calling reasoning in the industry, train of thought, or a more general idea of just "looking like it's smartly working through something and not just mimicking that", so I covered both case and mentioned that this is tuned during RL.

What you just described as reasoning is the general case of reasoning, whereas we specifically mean it now to be a separate phase from normal output, where the model wraps its output in <think> tokens etc (depending on implementation) and in some cases, as with frontier models, even hides and summarizes what is actually being generated during this step. During training, the models learn to exhibit differentiation in modality depending on if they're in a reasoning or regular output phase, so it's not just "loop the model back on itself but call it reasoning".

loading story #50548937