Hacker News new | past | comments | ask | show | jobs | submit
jakswa | on: Muse Glimmer: 30B-parameter model optimized for always-on local agent workflows
Another candidate for the 7900XT (20GB VRAM) I got sitting around. I pulled latest llama.cpp (targeting vulkan during build) after seeing a muse PR merged a few hours ago, and unsloth/Muse-Glimmer-30B-GGUF:UD-Q4_K_XL runs on my 7900XT barely (and with no MTP). Sits at 19GB VRAM w/ 4 parallel 113k context slots, all layers on GPU, and at 700 tok/s prompt, and ~36 tok/s generation.

Waiting on Q3 to download to check speed + do my usual anecdotes. I generate beefy code snippets and poems, and also ingest my HOA declaration and answer nuanced questions.

edit: i should've prefaced this somewhere with: This card ballparks at 800GB/s IO, which I can't seem to find easily on the market anymore. Kinda the ideal card for this model, if I just had a _little_ more VRAM (XTX is 24GB).

edit2: not mtp, this is dflash model (param in child comment). I'm up to ~60 tok/s generation and sitting at 19GB VRAM (i added --no-mmproj (makes it text-only i believe) because I'm used to speculative decoding wanting more VRAM and I'm already close to the limit :sweat_smile:)

jakswa | parent
Q3 results: unsloth/Muse-Glimmer-30B-GGUF:UD-Q3_K_XL gets down to 15.6GB VRAM and full context (131k) on the 4 parallel slots. Prompt/generation speeds about the same. Overall feeling like a nicer-fitting Qwen 3.6 27B, but want to test out MTP generation speeds once I can.

edit: My favorite bit of reasoning I saw go by in my "generate me a beautiful code snippet" anecdote: 'Could give a snippet of beautiful code: the "hello world" in brainfuck? No.'

edit2: my first dflash speculative model! no mtp. I'm up to ~60 tok/s on empty context with `--spec-type draft-dflash`