Hacker News new | past | comments | ask | show | jobs | submit
cynicalsecurity | on: Muse Glimmer: 30B-parameter model optimized for always-on local agent workflows
I don't understand the desire to run own AI models for programming locally. No laptop is ever going to be as powerful and energy efficient to run anything close to OpenAI, Anthropic or Google models. A model you can run on a loptop is simply not going to work as well as it's needed for programming. Small models for linguistic work fine, but anything more sophisticated simply won't provide enough resources or power. Or models would need to be significantly dumbed down - then why use them at all? So far the idea of carrying a "thin" or "thin"-like device looks more reasonable to me, while running AI on your own server.
OtherShrezzing | parent | next
> A model you can run on a loptop is simply not going to work as well as it's needed for programming

The models you can run on a high-spec laptop today are approximately where frontier models were 12-18mo ago (albeit at a lower tok/s rate). If you scan back through hn comments from that era, you’ll find plenty of people saying “this is powerful enough to massively increase my productivity”.

anon373839 | root | parent | next
> albeit at a lower tok/s rate

Not always! I get 80-100 tok/s from Qwen 3.6 35B-A3B on a MacBook Pro thanks to MTP. With long contexts that dips to around 50-60. However, prefill is much slower than API models. So it becomes really, really, really critical to not have cache misses.

drivebyhooting | root | parent
Slower is meaningfully dumber when you’re time bounded and need all the inference time compute you can get.
linguae | parent | next
I’m quite optimistic about the long-term future of local LLMs for privacy and cost control reasons. An LLM running on my own hardware, even if it’s not a laptop but a home server, is one where I don’t need to worry about token limits, token fees, privacy, and “rug-pulling” from the vendor.

In the short term, the big challenge is being able to afford hardware that can run a ~30B model. Last month I got to experiment with LLMs on a NVIDIA RTX 6000 Ada Generation as a visiting researcher during my summer break. I see the power of local LLMs for agentic coding; they’re no Claude, but they are quite useful. I wish I had gotten into local LLMs before hardware has gotten prohibitively expensive and in some cases unavailable; Apple discontinued certain Mac Minis and Mac Studios with high amounts of RAM due to the RAM shortage.

Hopefully high RAM prices don’t become a new normal, though the next year or two doesn’t look good.

ComputerPerson | parent | next
I've never done it but would be interested because it cuts out the burden of worrying about costs. Maybe I'm mistaken on energy cost here. There's a constant raincloud that follows me around regarding limits, and it would be nice to shake that.

I've been able to accomplish incredible feats (for myself) since GPT-4, so model intelligence is secondary.

lluisantoni | parent | next
For some companies there might be a need to run them locally. For instance, Apple decided to run LLMs on the phone locally. I guess it depends on how important latency and privacy are. Perhaps Meta is looking at how much interest for those local models is there.
brandon272 | parent | next
> I don't understand the desire to run own AI models for programming locally.

Privacy. Security. Not bulk uploading your trade secrets and intellectual property to Sam and Dario’s servers.

cynicalsecurity | root | parent
I was suggesting running personal AI servers.