Thanks for posting! Have you had any success with running without kv cache quantization? Is there a noticeable difference in quality without any? I would assume that would eat into context but 170k is pretty generous!

I'm hoping too that they'll put out some MoE variants.

Qwen3.5:122b:a10b can run about twice as fast as this 27b dense model.

Edit: Like its predecessors, 3.8 seems really inclined to overthinking, and on a 27b dense model that's kind of painful. I think I'm going to stick with gemma4:26b-a3b as my go-to because it runs about 4x as fast and tends to only need a fraction of the tokens in its 'thinking' stage to get the same or similar answer.

Me too. 35B A3B runs really fast on my MacBook Pro (M4 Max) and is suitable for real-time tasks like dictation post-processing. The dense model is not.

Same here! Qwen3.6-35B-A3B is the only local model I've found that runs reasonably on my iGPU. Looks like me and and my noisily-wheezing laptop will be sitting out this upgrade.

> I'd gladly take A5B or A8B or even A10B as a sort of middle ground.

Whats up with focusing on the active param count? Do yall fiddle with the weights or something?

> Beats Opus 4.7 Max

I'm a huge open model fan, and have used them since forever, even have daily drivers for on-prem dev, but no. They do not beat opus on real-world usage.

Qwen models are impressively good for what they are, are "good enough" for plenty tasks, can be ran locally on decently priced hardware, and so on. They certainly have their uses, and the field in general has advanced faster than my early expectations. But to compare a 27B model to SotA behemoths from a few months ago is doing everyone a disservice, especially people who pick it up, try to use them just like API models, and leave disappointed and confused. Number goes up on a benchmark isn't it.

Is there any advantage to using the model from Unsloth compared with https://huggingface.co/Qwen/Qwen3.8-27B-FP8 ?

Considering the clusterfuck that is opus 5 or even fable, if Qwen 27B is trully better than Opus 4.7 Max, I will rejoice.

That's crazy, considering the massive size difference. But the small Qwen models are known for punching above their weight.

I wish each quant was benchmarked on the same tests as the original network so we could compare their performance

Good morning Dario!

I'm convinced a lot of the anti-open-weight model comments at this point are inorganic traffic - there's trillions in investor money riding on a world where these models aren't cheap commodities. Having actually used things like the recent GLM, Kimi, and Qwen I think any edge the labs have is marginal at most and actually prefer the open weight models in most day to day usage.

Anthropic's recent releases are wordy to the point of exhaustion. Every time I use opus recently I find myself wanting to yell "GET TO THE POINT" at a terminal, which is exacerbated by it being slow.

> I've settled on GLM-5.3 (formerly Deepseek v4 pro 0813) for architecting

Dude, GLM-5.3 released _today_.

The phrasing "I've settled on" is incorrect for this context.

i think you will like luna if you haven't tried it yet

I could kiss you right now

Sounds like you need to check what the max context is set to ...

Use 3.6 27b as a daily driver for months with charmbracelet crush. Gemma 26b-A3b is not even remotely comparable in terms of coding for me. YMMV depending on how you work, what harness you use, etc I suppose.

gemma4 can't hold a candle to 3.6

You can add a thinking budget thats not much effort in llamacpp. You can align the cut off message with your agent instructions.

What you describe is a engineering harness problem.

If you, and i mean the royal you, actually read tge thinking traces you can see and figure out where its stuck

This means an effective harness would observe when the model is overthinking and step in with reasonable redirection, like increasing logging.

Llamacpp can set reasoning budget and message per reauest, so it can be dynamic.

Your complaint is "skill issue" based and will be resolved by people who do something ither than vibe code react demos.

Go for a slightly more quantised version, and experiment with different MTP settings. I find that MLX versions are marginally faster on my 64GB M1 Max, but I usually use Unsloth's GGUFs via llama.cpp as there's a much greater range of quants available and I prefer llama.cpp. MTP sometimes also helps a little, but I suspect it's less helpful on my system than others.

Unsloth: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

This might work for you, but I didn't get on very well with MTPLX when I tried it a while back; YMMV: https://huggingface.co/Youssofal/Qwen3.8-27B-MTPLX-Optimized...

I have an M4 Max (unfortunately 64GB). I have been running the Qwen 35B A3B one for a while now, after testing and benchmarking a number of models. That one was consistently the best in class for tasks like despamming, E-mail classification, OCR and dictation post-processing. It was also really fast (90 tokens/s).

I'm benchmarking the 3.8 model now, it seems it is better (near-perfect score on my E-mail spam filtering benchmark, best of any model I tested, ever). But it is slow.

One thing I would recommend is keeping an eye on MTP parameters. I tested and benchmarked extensively, and I use `--spec-draft-n-max 2` with llama.cpp. Longer sequences actually decrease overall performance.

As for running, I ended up using llama.cpp and its llama-server, with a bunch of scripts written by AI, because I got tired of LM Studio not implementing the image-related parameters which made gemma4 useless for OCR.

27B is a dense model so it will be slower with an MoE (A3B), but should have better quality? I still haven’t found very good uses cases on my M3 Max for dense models. Even if you can find a MTP version, it doesn’t help much, especially if you compare against an MoE with MTP as well.

Wait for the MTP variants that will likely be out within days. I'm on a 128GB Strix Halo box and for 3.6-27B 8bits I was getting about 9tok/sec (not great). With MTP that gets closer to 18 tok/sec (kind'a usable).

Unfortunately, that chip just doesn't really have the memory bandwidth to run this (or nearly any) model at acceptable speeds. I have the exact same chip (M4 Max 128GB) and I've been trying to optimize a completely purpose-built implementation with Fable and this is just not possible. Even if you could reach the full 576GB/s, it's just physically impossible to exceed these numbers with the model's architecture:

2 bpw - ~85.7t/s

3 bpw - ~58.0t/s

4 bpw - ~43.9t/s

6 bpw - ~29.5t/s

8 bpw - ~22.2t/s

16 bpw - ~11.2t/s

without cheating. You'd have to exclude layers, skip operations, etc. basically do stuff the model wasn't trained for. And speed collapses so fast with context that even 2 bpw would be looking at ~37.6t/s after just 128K tokens.

MTP only improves the situation by up to 2x in the ideal case, while drastically reducing the performance floor. While optimizing a 9B model on this hardware, I've found that the GPU just doesn't have enough FLOPS to handle speculating more than one or two tokens ahead on a single stream, regardless of quant level, simply because of the arithmetic cost of the forward pass. The 27B model would be even more expensive than that, potentially such that it's already bottlenecked by the GPU itself rather than memory.

I wouldn't get my hopes up for the 35B-A3B either. Not only is it reportedly much less intelligent, but I hit a similar ~85t/s wall in practice (again with highly specialized inference).

Without speculation I can reach around 120t/s on Qwen3.5-9B and with n-gram speculation (not even MTP; this derivative didn't come with one) around about 150t/s on average. This is on the very very edge of what I'd consider acceptable for me to even consider using such a compact model. YMMV due to the silicon lottery but the situation isn't good.

Check out MTPLX and limit your context size.

I'm hoping for a 3.8-122B MoE

can you tell me ideas of usecases of 9 or 10B language models ? I cant find any usecases other than training a lora on them to give good bash commands for example

As was the case with GLM 5.3, it seems that there is still much juice to be squeezed from post-training

Was recently thinking about doing something similar, do you basically just have the qwens describe what they see for the flashes?

Was considering adding a LoRa/vision head to Flash, but seems like it could take a while to get it right.

If DSv4 Flash was multimodal, I’d probably be done model shopping for a while

Pretty sure you can use Gemma models on Google's "Vertex AI".

There is a down to earth guy on YT that performs a series of tests against LLMs running on non-god-tier commodity hardware. He will likely be testing this soon enough.

https://www.youtube.com/@lukesdevlab

I don't know if that is what you are looking for or not and as always your experiences may be different.

With spec decode should easily get to >100tps

on my dual 3090s qwen 3.5 27b was running at around 110tps using the config from https://github.com/noonghunna/club-3090

make that 200tps on a single 5090, 4x faster than opus https://x.com/radixark/status/2088285681131110446

devs about to get handed a two 5090 box each and told to max that out

strong visual reasoning apparently, which is nice. still lacking native audio however. hoping for more companies to embrace the spirit of something like `gemma-4-12b-qat` for actual multi-modality (text, image, video, audio).

'Good' news, there seems to be nothing new architecture-wise. Same as Qwen 3.5 and 3.6, so llama.cpp doesn't know the difference.