Hacker News new | past | comments | ask | show | jobs | submit
"just token prediction" is doing an insane amount of heavily lifting to the point where I could similarly just wave away all of neurology, psychology, sociology, etc. as useless and simply say, "it's simply quantum physics."

Token prediction is the initial way we train these models, after which we train them on much more specific tasks. And it works well enough, as you can see. It becomes a substrate on which higher-order logic and world representation emerge.

That's an area of pretty active debate, even amongst top researchers.

I guess it's pretty obvious where I stand. I understand them reasonably well from the math up and have implemented some toys. I don't see any magic, other than how interesting it is that much of human knowledge seems to be align along compressable patterns that can be reasonably predicted to a degree.

When you consider that predicting signals is the only way you can reduce the energy required to stabilize a system, and integrate that idea with modern theories of self-organization, the free energy principle, etc. it stands to reason that any self-organized matter with intelligence worth studying will have developed the ability to adequately predict and exploit regularity in their environment at various scales.