Two recent gems from the 6-series (Astra and Sol)... I get variations of these all the time:
- "Correct. My earlier 'yes' was inaccurate."
- "I used 'done' too broadly."
...You don't accidentally use "done" too broadly. You're either done, or you're not. And when the model hides caveats in its reasoning steps which clearly indicate it knows that it isn't done, or that the answer is actually no, it's hard not to interpret it as gaslighting or deceit.
This is magnified by how insanely defensive the 6 series gets, and how much gaslighting and deceit OpenAI themselves have engaged in with how much better the 6 series was supposed to be, and why it meant our subscription value should be cut in half. I don't take for granted how far LLMs have come in a few short years, but I'm being told that these models are to transform my entire industry.
You never trust an initial "done" from an LLM. Always ask it to double check and verify once a task is done.
I replied above, but I typically get a 2:1 or 3:1 ratio of refactoring to building prompts. For every "build X" I have to have 2-3 "fix Y" type of prompts, whether the fix is removing code-smells, duplicated code, off pattern code, etc.
Your advice might be helpful to literal vibe coders but it's not applicable to this discussion.
Maybe there's a harness or a prompt that can impart that onto agents, but if it hasn't happened yet, I become less and less confident that this entire branch of AI development isn't simply a dead end.
Then in the instruction tuning phase we try to get them to unlearn their fundamental nature and start to produce correct output instead, which works to some extent, but not perfectly.
I think it's only a matter of time until we adopt better pre-training methods that don't bake in this tendency to BS, and hallucinations will be a thing of the past.
> it's important to understand that it's literally just a tack-on
RL is not a tack-on, it's a critical part of training which affects a lot more than how "human-sounding" the output is. I'm not totally certain what you mean to express when you say "reasoning" since it has precise meaning in this context and you might be speaking about something more general.
> there is no internal drive for them to be "correct" or "accountable"
RL is meant to solve this. The problem is currently it has created internal motivations which are antithetical to accountability. We shouldn't take this as an axiom, a lot of smart people are trying to solve this.
> Maybe there's a harness or a prompt that can impart that onto agents
That's the issue, this needs to be fixed in training, the incentives are not aligned and that bleeds into its interpretation of system prompts and user directives.
Ultimately, the problem is not a lack of my understanding of the limits of today's frontier models. The problem is the companies peddling the models, these companies have perverse incentives to oversell their capabilities to please investors and feed corporate ego, even if it means gaslighting and deceiving me about said limits.
> RL is not a tack-on, it's a critical part of training which affects a lot more than how "human-sounding" the output is. I'm not totally certain what you mean to express when you say "reasoning" since it has precise meaning in this context and you might be speaking about something more general.
They never mentioned RL, it's "reasoning" they called tack-on, which is right. It's using the model's output as further input to simulate a train of thought and hiding that part from the user, but at the base it's still text prediction.
"Reasoning" started out as something users figured out, manually copying outputs to the next input to see what the models came up with, and the models would correct themselves without human input so it became automated.
What you just described as reasoning is the general case of reasoning, whereas we specifically mean it now to be a separate phase from normal output, where the model wraps its output in <think> tokens etc (depending on implementation) and in some cases, as with frontier models, even hides and summarizes what is actually being generated during this step. During training, the models learn to exhibit differentiation in modality depending on if they're in a reasoning or regular output phase, so it's not just "loop the model back on itself but call it reasoning".