Does it actually? Sure, it is operating at huge scale and this design choice allows enormous models (presumably enormous MoE models). But is it achieving economies of scale?

Economies of scale are rooted more in the idea of a profitable thing becoming broadly sustainably cheaper as the market grows, allowing cheaper manufacturing and further growing the market.

The big two or three LLMs are cheap because they are substantially subsidised by investor cash, and they are not profitable at all in any sustainable sense: the current pricing can only be sustained for as long as they can raise money, and the market will (and does!) shrink when it is asked to pay a less-subsidised price.

There appears to me to be no route to true profitability for this design strategy at all. Even the open weights router companies are downstream of largesse from state-affiliated companies.

Qwen flash-next is at least as good as Sonnet for my uses, and the speed is perfectly acceptable.

I really think the big labs are going to soon start feeling pricing pressure for everything smaller than Opus-tier models, and it will come for those next.

In the mainframe's heyday, it was difficult to grasp a world in which compute became so dense and so cheap that it made sense for each individual to have all processing and memory on their desk, talk less of on their lap and in their hand.

Of course this time could be different. But, when we look at previous relentless progress in compute and efficiency, then also couple that with the breakthroughs AI itself unlocks, it's hard for me to imagine that centralized AI will meet both the scale and flexibility desired for all use cases.

Just as centralized compute never fully disappeared -- it took the form of client-server computing and the cloud, there's no question that centralized inference will continue to exist.

But it wouldn't surprise me if, ten years from now, 90% of global inference happens at the client.

While simultaneously, the 10% of inference that takes place in the datacenter is wildly lucrative and orders of magnitude greater than today.

Put simply: the inference pie may become so large that both client and datacenter win.

What new bike should I get? How should I rephrase this email? What to do about my persistent headaches? Recipes based on these ingredients?

It'll become easier and more affordable to distill/train/optimize smaller local models to meet your specific work needs. Multi-agent efforts will orchestrate the balance between using on-prem hardware for most tasks, and then defer to cloud models for more demanding/edge-case tasks. Especially if an organization can manage what data is shared with which model, and most of their user base are everyday casual users.

I also expect cloud providers to eventually start charging different rates for on-peak and off-peak usage, which would incentivize running what you can with the local compute you have otherwise sitting idle across dozens to hundreds to thousands of machines in your fleet.

Anthropic just changed their policy to invalidate any unused credits without refund if you buy your API credits annually. I see that as a sign of unpredictable budgets for cloud models for customers.

I could be wrong and all your future hardware purchases for end-users will be thin clients, without the local processing capabilities of today's laptops and towers. I don't see the hardware manufacturers profiting much from that, but maybe we'll see more companies like AMD declaring they prioritize cloud compute customers over everything else.

With a hardware refresh for cloud computing every 3-5 years, we'll likely see today's compute hardware driving Opus and Sol available on the secondary market for companies willing to build more on-prem capabilities.