Let's not forget about Yann LeCun's current area of research that's completely different from LLMs: Joint Embedding Predictive Architecture (JEPA)
If he gets that style to be more efficient (they're already competitive) it'll completely kill off LLMs