It's the transformer architecture (token embeddings and cross-token encoding) and residual stream (cross-transformer encoding and token differentiation) that make LLMs powerful tools.

The key question is not whether this architecture is powerful. The key question is how well these models can represent things outside of their design.

With chess for example, are they actually learning the rules and strategies of the game (like AlphaZero/AlphaGo, Leela, etc.) or are they just relying on a large corpus of chess games? For example, a lot of early game moves will have been played out and the best moves will likely be the more frequent moves in the training set. So LLMs could do well early game based on that game corpus.