My main reasoning was that transformers was the lightning in a bottle and the best work is in extending it instead of transcending it, which requires you to capture another lightning . Which to me appears to miss the assignment. OpenAI, Antrophic, they understand this intimately. Google on the other hand, fell victim to their own ambition.
If your goal is purely commercial, or time critical, then a product-based approach of squeezing all the juice out of LLMs makes sense.
If your goal is truly human-level AGI then this is more of an open-ended research endeavor, and timelines are hard to predict. Arguably we have only "captured lightning in a bottle" once in the last decade - the original 2017 attention paper - and so the timeline for a "few more Transformer-level breakthroughs" might more realistically be estimated in decades rather than years. You could argue that the application of RL to LLMs as a training method was a second "lightning in a bottle" but I don't think it changes the expected timeline of such discoveries by much.
The time criticality seems to have become a huge factor for those pursuing LLMs, and certainly for OpenAI and Anthropic, who regard it as a race.
It seems absurdly obvious (though many would disagree!) that LLMs alone are not going to achieve human-level intelligence and cognitive performance (using a slightly broader term there to include things like creativity, for those that might not consider that as part of intelligence).
If you compare a Transformer to a brain, then the best parallel is that a Transformer is functionally similar - in being a prediction engine - to part of our cortex, but of course that means ignoring the other half our cortex - the feedback paths that enable continual learning, which in turn supports creativity.
Of course people will probably respond "you don't need flapping wings to fly", but if you want to fly you do need SOME way of doing it, so brain comparisons are still valuable... If you look at our brain architecture and identify all the components and connections that have no equivalent in a Transformer, and if the goal is human-level capability, then you do need to understand what each of those brain components achieve functionally, and have SOME way of providing that functionality in your LLM+ or whatever you call it. LLMs' lack of any functional equivalent to our cortex's feedback paths - lack of continual learning - has been recognized as one major functional deficit, but there are probably half a dozen others too, reflecting the multiple "Transformer level" breakthoughs that Hassabis notes are needed.
While you don't need flapping wings to fly, you do need to invent the airplane, and even after 100+ years of airplane advances we've yet to build an airplane even remotely as capable as what some birds and insects are able to achieve.
Maybe 2017 was the Wright Bros moment where humans first learnt to do some of what our brains can do.