> Would that imply the entry-level MacBook will have 128GB

Most people neither need nor want to run large LLMs locally.

The entry level MacBook Neo costs $599. Even if RAM prices revert to the pre-AI mean, the entry level models are not going to be that stacked with RAM.

Eventually we'll get to a doubling where entry level means 16GB of RAM, but most people won't be running anything more than simple models for light local work.

Have you ever tried running large LLMs on a MacBook Pro? It's hot, loud, and makes your battery level disappear at rates you didn't know were possible if you're not plugged in. It's a very niche hobby.

Even if it's 2030 - that's a mere 3 years from now - it will take at least that long for the greater economy to figure out how to use LLMs. And probably a memory boom/bust cycle in that time too.

If Apple, in 2030 has a compute product that a worker can BYO(X) to work that has 2TB RAM, with an encrypted personalized context that can float between agents, then they will be in a very interesting place.

The Mac Studio hardware can do 512GB. That might be a hardware limit of the current architecture.

But the family shouldn’t be limited by that. We’re not going to see M6 Max hardware, because the M7 line is being fast-tracked and pulled forward. And I wouldn’t be at all surprised to see M7 capable of 1-2TB or more.

> whether Apple can ship affordable consumer devices with enough memory to run sufficiently useful models sufficiently fast.

i suspect that they will at least do offloading for casual usage as well. if they can ship 24GB in a homeview or appletv style product, it's game over.

I'm on 128gb Mac and Laguna2.1 is able to do 800tps coding as good at Sonnet 4.5 or Opus 4. It's awesome.

Apple’s position down the road. When the Chinese company CMXT takes a much bigger share of the memory market worldwide, and they will. Their dominance in memory will probably be pretty close to what BYD is today in the worldwide car industry.

I don’t think Apple can be in a position where all of the Chinese device-making companies can get memory at a significantly cheaper price from CMXT, and Apple is stuck with buying memory from the three hyenas of memory at a significantly higher price.

That’s not going to help Apple sell their devices at a competitive price worldwide over time against the Chinese computer makers (Lenovo, Asus, and Motorola) and make no mistake about it: their share of the market worldwide will get bigger because of this change in the memory market; their focus will be on the long term, while the lawyers/MBA/finance types in the West will have their heads stuck in the short term.

It’s the (Chinese companies’) opportunity, and they’re not going to let it pass; I wouldn’t. In addition, Apple is in competition with Nvidia, Microsoft, Intel, AMD, Qualcomm, Google, Meta, and indirectly with Anthropic and OpenAI (whose CEO, Sam Altman, recently declared that computer ram is wasted on low-level regular computer users), along with a whole legion of other 3D model makers on the procurement of memory.

The combination of all these elements going into the future means that if Apple wants to stay ahead, I believe Apple has no choice but to take memory in house, they have the design and engineering capability in-house, and they have the money because they didn’t burn it on AI models and data centers.

This scenario is no different than replacing Intel or Qualcomm; you can see the writing/destination and the ultimate train wreck up ahead if Apple doesn’t roll up its sleeves and get busy, the market has changed and again Apple has to change and adapt to the new conditions.

Most likely Apple has been spending whatever they can to get in front of the memory market.

I dunno, I reckon they are targeting 64GB and will develop a smaller model thats built in by default that handles a lot of simple tasks quickly.

Probably sitting under a router, so that it can hand off bigger jobs to an Apple DC somewhere, probably with a certain number of credits based on your iCloud sub.

Why is there a desire to have this stuff on the edge on the engineers lap instead of just building some arbitrarily scalable cluster for I'm guessing cheaper than apple's usual premium? Habitual practice from the engineers who would compile on the device getting hot in their lap? Lack of widespread hpc experience compared to running docker?

Why do you need huge economies of scale though? For 90%? of queries a local model will be good enough?

Most people just Google something or use the free ChatGPT. They have no clue how the answers get there and frankly would be just as pleased with GPT 4o as they would be with Astra. Questions like “what does this light mean on my dashboard” or “which bidet is best”.

My guess is it will probably be both. There was a time when no one could envision computing without the mainframe.

In the mainframe's heyday, it was difficult to grasp a world in which compute became so dense and so cheap that it made sense for each individual to have all processing and memory on their desk, talk less of on their lap and in their hand.

Of course this time could be different. But, when we look at previous relentless progress in compute and efficiency, then also couple that with the breakthroughs AI itself unlocks, it's hard for me to imagine that centralized AI will meet both the scale and flexibility desired for all use cases.

Just as centralized compute never fully disappeared -- it took the form of client-server computing and the cloud, there's no question that centralized inference will continue to exist.

But it wouldn't surprise me if, ten years from now, 90% of global inference happens at the client.

While simultaneously, the 10% of inference that takes place in the datacenter is wildly lucrative and orders of magnitude greater than today.

Put simply: the inference pie may become so large that both client and datacenter win.

Normie queries can be handled quite well by Haiku 4.6 or whatever.

What new bike should I get? How should I rephrase this email? What to do about my persistent headaches? Recipes based on these ingredients?

Unless cloud PCs take off and kill "AI PCs" and the future of the end-user hardware market, I think it'll be a bit of both.

It'll become easier and more affordable to distill/train/optimize smaller local models to meet your specific work needs. Multi-agent efforts will orchestrate the balance between using on-prem hardware for most tasks, and then defer to cloud models for more demanding/edge-case tasks. Especially if an organization can manage what data is shared with which model, and most of their user base are everyday casual users.

I also expect cloud providers to eventually start charging different rates for on-peak and off-peak usage, which would incentivize running what you can with the local compute you have otherwise sitting idle across dozens to hundreds to thousands of machines in your fleet.

Anthropic just changed their policy to invalidate any unused credits without refund if you buy your API credits annually. I see that as a sign of unpredictable budgets for cloud models for customers.

I could be wrong and all your future hardware purchases for end-users will be thin clients, without the local processing capabilities of today's laptops and towers. I don't see the hardware manufacturers profiting much from that, but maybe we'll see more companies like AMD declaring they prioritize cloud compute customers over everything else.

With a hardware refresh for cloud computing every 3-5 years, we'll likely see today's compute hardware driving Opus and Sol available on the secondary market for companies willing to build more on-prem capabilities.

And if you're on a mobile device, any halfway powerful local model is going to cut your battery life way down.

Yeah totally agree

A few years back we never thought open source llm would get as good and even the small and cheap model

Its good to have a race and such competition

For those of using AI day to day, not being beholden to the whims of Anthropic and OpenAI is definitely a dream scenario.

Is Apple Intelligence not local models currently running on iPhones?

You don't have to sell it as a local model. Sell it as offline capabilities. Sell it as private background processing while your phone is idle and plugged in. Sell it as un-enshittified LLM once Anthropic and OpenAI inevitably screw everybody.

The design and engineering of memory is not more difficult than replacing Intel and Qualcomm, Apple certainly has the money, and they certainly have the engineering talent within the Apple Silicon design group. It took 13 years to replace Intel and six years to replace Qualcomm who has already warned their investors that they anticipate losing revenue in the near future. The question is whether or not they have the will to do so, money and engineering ability isn’t the problem, it is simply a matter of will. With CMXT coming online bigger and stronger.

I don’t think there will be a choice long-term for Apple the conditions of the market have changed, and you have to change with it or get left by the wayside. They have shown that they will adapt and they certainly haven’t wasted money like their tech competition over the years.

The contrast I'm drawing is running these models locally vs on the cloud and how that would be a selling point well received.