Hacker News new | past | comments | ask | show | jobs | submit
spmurrayzzz | on: Qwen 3.8 27B
> They do not beat opus on real-world usage

We have an internal eval that measures performance on tasks for a handful of embedded systems repos for our mmWave radios (mostly Rust, some C for microcontroller stuff). Qwen3.6-27B scores only 4% lower for pass@1, n=250 compared to Opus-4.8.

For the labeled dataset, the average PR size they're being measured against is around 1.5k SLOC.

This is very much "real-world usage" for us. The sort of change sets that come in daily/weekly and are solving non-trivial issues in the respective codebases.

As is usually the case, the most broad claims from both the labs and from the consequent pushback are talking past each other.

cyanydeez | parent | next
Let us know when you have Qwen vs Qwen comparison stats. As long as there's not a regression, that'd be awesome.
spmurrayzzz | root | parent
4% is within the margin of error anyways for pass@1, so I think pass@k > 1 is gonna be the better indicator of any movement (still need to calibrate the optimal k to re-test). 10 seems too tolerant even though that tends to be the next tranche I reach for.
croemer | root | parent
Depends on where you sit on the binomial curve. At p=0.04 for n=250 4% points would not be within margin of error.
enraged_camel | parent | next
>> We have an internal eval that measures performance on tasks for a handful of embedded systems repos for our mmWave radios

Okay but the parent said real-world usage, presumably meaning coding tasks.

We have a whole bunch of complex evals that Haiku 4.5 passes. That doesn't mean it is a good model for coding.

graceful6800 | root | parent | next
They literally stated in their first sentence that it was coding tasks.
spmurrayzzz | root | parent
Yes, these are coding tasks in the embedded systems domain (I mentioned Rust and C).
croemer | parent
How much does it score though? 0% would be 4% less if Opus was at 4%. Unless you mean relative fraction not percentage points - but people usually mean percentage points in such situations.
tyre | root | parent
0% is not 4% less than 4%, that would be 3.84%.

0% is 4 percentage points (pp) less than 4%.