128k context window is a complete non-started for us. We need to optimize our most needy agentic jobs, but our average context is well above that
I was able to use rope/yarn scaling with llama.cpp to extend the context window to 256K and it seems to be pretty usable on the debugging-and-bugfixing session I have that’s 216K tokens deep. No infinite output loops, reasoning is still coherent, tool calls appear to be passing and failing at roughly the same rate as a fresh context window. Haven’t tried going to 512K or higher yet but 256K definitely seems usable to me on a single Radeon R9700.
I added these arguments to my llama-server call, using the official GGUF release from Meta’s account on HF.
`--rope-scaling yarn --rope-scale 2 --yarn-orig-ctx 131072 --override-kv muse-glimmer.context_length=int:262144,dflash.context_length=int:262144`
Useful stuff, thank you. And more reading for me to do!
(DFlash seems rather weird on my M1 Max; model performance suddenly drops off a cliff like maybe there is some memory management issue).
In my experience I am getting 23-24 t/s output with dflash off, and it craters to ~9 t/s with it on, miss rate exceeding 50%. And I'm using the same device as stated on their model page/card. We might need to wait for the software to catch up
As an extra data point, I have exactly the same issue on a M1 Max as well.
The model card does say 131K+ but I have no idea what scope the + really has in practice. Feels like overclocking; you're-on-your-own territory?
One thing I will say is that its thinking traces are really quite terse. It genuinely seems to spend many fewer tokens on reasoning. So that might help a bit.