The frontier labs will do well if they pivot their offering towards more capable, larger-scale models that are inherently harder to both train and deploy for commodity suppliers. Their existing investments in gigawatt-scale datacenters are quite optimal for this. "Commodity" inference need not comprise the whole market.
I don’t think this works, for a few reasons. First, intelligence gains from scaling the models bigger is sublinear now. So they could eke out a little extra performance, but the increased cost will eventually eclipse the economic value gained from this.
Second, humongous models are impractical even for them to deploy widely. They’re best used as teachers for smaller, more efficient models that can crank out the volume they need to sell.
Finally, there is a data wall. Sure, they can keep scaling RL on math problems and code. But with everything else, where will the supervision come from when they need several orders of magnitude more?
I agree. And even if they were able to do it for one more round, it's not a sustainable strategy. What they (Anthropic and OpenAI) need to do is build platforms and integrate verticals.
assuming the technology of model architectures does not gain any further breakthroughs that returns us back to the gains previously seen. I'm of the opinion that we still have some discoveries on the mathematical side of the fence to go that will improve models further.
loading story #49298664