Hacker News new | past | comments | ask | show | jobs | submit
scrlk | on: Qwen 3.8 27B
Beats Opus 4.7 Max (w/ Claude Code) on DeepSWE (42.2 vs 40). Looks like Qwen's 27B models continue to pack some punch.

Unsloth's GGUF quants are up: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

NitpickLawyer | parent | next
> Beats Opus 4.7 Max

I'm a huge open model fan, and have used them since forever, even have daily drivers for on-prem dev, but no. They do not beat opus on real-world usage.

Qwen models are impressively good for what they are, are "good enough" for plenty tasks, can be ran locally on decently priced hardware, and so on. They certainly have their uses, and the field in general has advanced faster than my early expectations. But to compare a 27B model to SotA behemoths from a few months ago is doing everyone a disservice, especially people who pick it up, try to use them just like API models, and leave disappointed and confused. Number goes up on a benchmark isn't it.

spmurrayzzz | root | parent | next
> They do not beat opus on real-world usage

We have an internal eval that measures performance on tasks for a handful of embedded systems repos for our mmWave radios (mostly Rust, some C for microcontroller stuff). Qwen3.6-27B scores only 4% lower for pass@1, n=250 compared to Opus-4.8.

For the labeled dataset, the average PR size they're being measured against is around 1.5k SLOC.

This is very much "real-world usage" for us. The sort of change sets that come in daily/weekly and are solving non-trivial issues in the respective codebases.

As is usually the case, the most broad claims from both the labs and from the consequent pushback are talking past each other.

loading story #49301443
loading story #49301866
loading story #49301118
redox99 | root | parent | next
Yep. These small models are actually worse than GPT 3.5 at some tasks (like recalling facts). You can definitely make models smarter at specific tasks (like tool calling, coding) but you can't compress the entire human knowledge into a 30GB file. It's just not enough bits.
loading story #49301840
KronisLV | root | parent | next
> ...but no. They do not beat opus on real-world usage.

I agree, but then we just need meaningful benchmarks that clearly show that! Otherwise it's hand waving about something that should be put on paper in quantifiable terms.

loading story #49300608
loading story #49300222
loading story #49300861
loading story #49300233
loading story #49300296
metadat | root | parent | next
How can you say this when you haven't even tried it yet? Is it just hypothetical vibes?
willcmcc | root | parent | next
There is 0 shot you can make that claim about this model you have not used or downloaded yet
altmanaltman | root | parent
"Benchmark is stupid" and "model beats model on benchmark" are two different things, though. The second one is objectively true regardless of your views on the first one, right? To expect everyone to share your opinion that benchmarks are stupid is pretty weird, and just saying "no" to an objective truth is the definition of delusion.
jrflo | parent | next
That kind of result makes me suspicious of benchmaxxing. Qwen 27B is 100x smaller than Opus 4.7. Is it really 100x more parameter-efficient? Two orders of magnitude is hard to believe. I don't have the hardware to run a 27B, but I'm curious what real world use is like. Maybe I'll have to buy some usage on a cloud provider to run my own tests, but this seems fishy to me.
loading story #49301723
nblgbg | parent | next
Is there any advantage to using the model from Unsloth compared with https://huggingface.co/Qwen/Qwen3.8-27B-FP8 ?
ycui7 | root | parent | next
if you have the VRAM, use offical release. quantized model lose focus after long context and can do damages or thinking loop
benxh | root | parent | next
Depends on what software/hardware you'll run it. GGUFs from Unsloth can run on pretty much every single potato; full weights need beefy gpus
satvikpendem | root | parent | next
Unsloth usually also fixes the models when they bork something, which always happens. For Gemma for example the tool calling wasn't working for the longest time.
loading story #49301799
petu | root | parent | next
Unsloth one is gguf for llama.cpp (and some other on-device engines).

So advantage is not having to produce your own quantisation / gguf from .safetensors you've linked.

4chandaily | root | parent
Run the unsloth if you are using llama.cpp (GGUF)

Run the one you linked if you are running vllm (safetensors)

Foobar8568 | parent | next
Considering the clusterfuck that is opus 5 or even fable, if Qwen 27B is trully better than Opus 4.7 Max, I will rejoice.
UncleOxidant | root | parent
If it's as good as Sonnet 4.6 for most things I'd be happy.
edg5000 | parent | next
That's crazy, considering the massive size difference. But the small Qwen models are known for punching above their weight.
WithinReason | parent | next
I wish each quant was benchmarked on the same tests as the original network so we could compare their performance
scrlk | root | parent
Unsloth publishes KL divergence numbers which measures how much the quantised probability distribution changes vs unquantised: https://unsloth.ai/docs/models/qwen3.8#quantization-analysis

It's a bit bare at the moment, I assume they are going to add further detail later (eg comparison to other quants), similar to their other releases.

loading story #49301120
loading story #49301503