> The key: since every sequence has a unique latent rule, the model must infer that rule in-context to predict what comes next. This in-context learning ability underpins many of the key reasoning capabilities observed in language models.

This is a remarkable paper. This is the first time I've heard someone training the actual thing we're trying to get this stuff to do!

---

> This raises a radical question: Is natural language the only path to intelligence?

Of course not! We have octopi, ravens etc., which in many domain display higher intelligence than frontier AIs.

"Embodied reasoning" (genetic algorithm brute force solving physical tasks for a billion years, to name one solution) is definitely one very practical form of intelligence, although we're taking some shortcuts in replicating it.

I'm wondering if simplified analog tasks like Box2D puzzled would help too (or perhaps even simpler? Hanoi? Block worlds?). I know many companies are using simulations of 3D worlds for that.

What I don't understand is how that can integrate with the LLM (physical intelligence would seem to require specialized circuitry, if only for the latency). But maybe once we have good specialized models, LLMs can be trained on their synthetic data?