Hacker News new | past | comments | ask | show | jobs | submit
gpsx | on: Chain-of-thought can hurt performance on tasks where thinking makes humans worse
I saw an LLM having this kind of problem when I was doing some testing a ways back. I asked it to order three fruits from largest to smallest. I think it was orange, blueberry and grapefruit. It could do that easily with a simple prompt. When the prompting included something to the effect of “think step by step”, it would try to talk through the problem and it would usually get it wrong.
loading story #42003959
ajuc | parent | next
It's not thinking, it compressed the internet into a clever, lossy format with nice interface and it retrieves stuff from there.

Chain of thought is like trying to improve JPG quality by re-compressing it several times. If it's not there it's not there.

Jerrrrrrry | root | parent | next

  >It's not thinking



  >it compressed the internet into a clever, lossy format with nice interface and it retrieves stuff from there.
Humans do both, why can't LLM's?

  >Chain of thought is like trying to improve JPG quality by re-compressing it several times. If it's not there it's not there.
More like pulling out a deep-fried meme, looking for context, then searching google images until you find the most "original" JPG representation with the least amount of artifacts.

There is more data to add confidently, it just has to re-think about it with a renewed perspective, and an abstracted-away higher-level context/attention mechanism.

danenania | root | parent | next
> Chain of thought is like trying to improve JPG quality by re-compressing it several times. If it's not there it's not there.

Empirically speaking, I have a set of evals with an objective pass/fail result and a prompt. I'm doing codegen, so I'm using syntax linting, tests passing, etc. to determine success. With chain-of-thought included in the prompting, the evals pass at a significantly higher rate. A lot of research has been done demonstrating the same in various domains.

If chain-of-thought can't improve quality, how do you explain the empirical results which appear to contradict you?

loading story #42010453
| root | parent | next
{"deleted":true,"id":42007314,"parent":42005778,"time":1730385200,"type":"comment"}
easyThrowaway | root | parent | next
I have no idea how accurate it actually is, But I've had the process used by LLM described as the following: "Think of if like a form of UV Mapping, applied to language constructs rather than 3D points in space, and the limitations and approximations you experience are similar to those emerging when having to project a 2D image over a 3D surface."
Eisenstein | root | parent | next
These kind of reductive thought-terminating cliches are not helpful. You are using a tautology (it doesn't think because it is retrieving data and retrieving data is not thinking) without addressing the why (why does this preclude thinking) or the how (is it doing anything else to generate results).
loading story #42008663
bongodongobob | root | parent
Maybe if you bury your head in the sand AI will go away. Good luck!
lucianbr | root | parent
This is basically a reformulation of "have fun staying poor!". Even contains the exclamation mark.

Those people sure showed us, didn't they? Ah, but "it's different this time!".

loading story #42007723
loading story #42007533
loading story #42004007