> --cache-type-k q8_0 --cache-type-v q8_0

In my tests, even Q8 quantization for the KV cache comes with notable drops in performance for longer tasks. It does provide more context length in limited RAM budgets, but the longer context tasks are where KV quantization starts to show problems. It’s basically unnoticeable for simple and short tasks.

> --spec-draft-n-max 5

5 is a lot of tokens to draft. Are you really seeing acceptance rates to support that? When I tested it, 2-3 was the peak. Anything more started reducing performance except on highly predictable short outputs.

Thanks for posting! Have you had any success with running without kv cache quantization? Is there a noticeable difference in quality without any? I would assume that would eat into context but 170k is pretty generous!

Lol at that command. Why is this stuff so hard to run locally? I've spent a few days trying to figure it all out and haven't been able to. LM Studio doesn't work behind proxies. Ollama is confusing and doesn't seem to support Qwen3? And Llama.cpp is your command.

I just want to run `<some-command> <model-name>` with some default parameters set and for it to run locally.

Do you find it useful or worthwhile to split a large LLM across two GPUs on a desktop?

If you've tried it, what worked well and what didn't? I'm especially interested in mismatched VRAM setups, e.g. a 16 GB GPU + a 24 GB GPU.

How much overhead did you see from inter-GPU transfers, and did the extra usable VRAM outweigh the performance hit?