Hacker News new | past | comments | ask | show | jobs | submit
mark_l_watson | on: Muse Glimmer: 30B-parameter model optimized for always-on local agent workflows
Meta is rocking AI. As of last week I have been using their excellent muse coding harness with their model Muse Spark 1.2.

Starting this morning I am running their new local 30B model muse-glimmer on my old MacMini 32G using Ollama (remember to increase the context size!) and pi coding harness. I am getting good results with muse-glimmer running locally, with the caveat that everything runs slowly (e.g., give it a task and then go walk outside or do Qi Gong exercises for a while).

cube00 | parent | next
Friends Don't Let Friends Use Ollama https://news.ycombinator.com/item?id=47788385
LeBit | root | parent | next
When that article was first published , I started looking into llama.cpp. With the help of an LLM I researched the knobs to turn that made most sense.

Things went from "local models are useless unless you have a 512GB GPU I guess" to "oh.. I can do a lot of stuff locally now!"

chorizo | root | parent
Had a similar experience. Llama.cpp compiled natively; parameter sweep to find best options fitting my use case for the qwen models with 16GB VRAM. The whole thing packaged into a portable container.
mark_l_watson | root | parent | next
From the Ollama docs for this new model:

Ollama's MLX engine provides state-of-the-art performance on Apple Silicon, with support for DFlash and image input:

ollama run muse-glimmer:30b-mlx

lenerdenator | root | parent
What do you use instead?
SwellJoe | root | parent | next
If you need a GUI, Unsloth Studio and LM Studio are both great frontends for llama.cpp. If you don't need a GUI, llama.cpp is the business for single-user deployments. Easy to use, always gets new model support very quickly, built-in Hugging Face client/cache support, works on probably everything (Mac, ROCm, CUDA, Vulkan, etc.).
computershit | root | parent | next
I use llama.cpp w/ llama-swap

https://github.com/ggml-org/llama.cpp https://github.com/mostlygeek/llama-swap

loading story #49249140
LeBit | root | parent
Thanks for the link to llama-swap. Didn’t know about it and will definitely install it.
reilly3000 | root | parent | next
try oMLX or vMLX - both great projects that offer some amazing performance optimizations for Apple Silicon that utilize UMA and NVME caching efficiently.

https://omlx.ai https://vmlx.net

That said, it's been a few weeks since I've looked so maybe llama.cpp has those features now... they really do move that quickly.

scotty79 | root | parent | next
Jan is so much better.
soupspaces | root | parent | next
The link in that discussion has a section about that
khimaros | parent | next
seems to underperform on Terminal Bench compared with qwen3.6-27b: 51.7 vs 60.7
mark_l_watson | root | parent
To be honest, I never give benchmarks a look. I just use the models for whatever I need to work on, so I can't really make comparisons that are useful for other people.
| parent | next
{"deleted":true,"id":49247706,"parent":49244699,"time":1786386534,"type":"comment"}
spaceywilly | parent
Newb question but I’m curious what would help it to run faster? Would it need more vRAM or just system memory?
spmurrayzzz | root | parent | next
The biggest gain you'll get is faster memory, provided you have enough capacity to load all the weight into vram. The DGX sparks and Apple silicon memory bandwidth (and also memory access latency) drag down the decode speed quite a bit.

I have two GPU rigs both with 2x RTX Pro 6000, can get ~250 tk/s decode with deepseek-v4-flash in native mixed precision. For context, in antirez's dwarfstar project he only gets ~20-40 tk/s on the same model @ 2bpw on M5 Max.

The latter is for sure usable if it's your only option, but it's really hard for me to personally go back to speeds like that when I've experienced the former.

(Also worth noting dwarfstar only has experimental support for dspark spec dec, when that lands it will definitely give a big boost at higher acceptance rates)

wincy | root | parent
It runs very quickly on my RTX 5090 fwiw. Whole thing is loading entirely into vRAM with a ~130k context size (the max) fitting as well.
codazoda | root | parent
That's a $5k 32GB card for anyone who doesn't know all these off the top of their heads (like myself).