Hacker News new | past | comments | ask | show | jobs | submit
vikramkr | on: Five Qwen3.8-27B models match Fable 5 Using GVS5H Harness
So using this harness five qwen3.8-27b models roughly matched Fable 5 on a saturated coding benchmark from early 2025. No comparison to astra/6.1 sol/fable 5.1/opus 5.5. No comparison on a benchmark that reflects actual agentic work (and agentic behavior is what modern models optimize for anyway - probably makes qwen look worse than it is?). And also obviously using a very old benchmark like this:

> The updated release of the dataset with problems released between May 2023 and Apr 2025 containing 1055 problems

means essentially guarenteed contamination.

I have no idea if this harness is any good. Agressive orchestration could be interesting given how hard models are RLed on orchestration? But the methods used to evaluate this harness are not informative

binsquare | parent | next
I do not think we should trust this harnesses claims.
R1ck404 | root | parent
I don't think we should trust 'small model beats giant frontier model' either... Don't get me wrong I think small models are and could be great, but matching something like fable 5?
| parent | next
{"deleted":true,"id":50228421,"parent":50228389,"time":1792903659,"type":"comment"}
hirako2000 | parent | next
Also why not comparing harnessed Qwen to harnessed other LLMs.
chvid | root | parent
I guess one point is you can run Qwen on a local machine.

It is not public knowledge how Fable 5 works - it is all hidden away behind a web service - for what it is worth it might a hybrid architecture with harness, loops, tools etc. combined with a traditional llm like what is done here.

hirako2000 | root | parent
Fair points. But harness at this point makes more difference than using the best model.
stymaar | parent
The other criticisms are valid but this one is too much:

> No comparison to astra/6.1 sol/fable 5.1/opus 5.5

I mean, should a 27B model beat Fable 5 but not Fable 5.1 that would still be an incredible achievement.

vikramkr | root | parent
The problem is that it's a very simple, older benchmark, and qwen was trained after fable 5. You need to test against similarly modern models so you aren't putting the models on uneven playing fields in terms of contamination. And for the stuff qwen is trained to be good at (agent stuff) opus 5.5 is light-years ahead of fable just because it's a newer model amd they RLed on it.

Though point taken that in a case where they didn't even bother making sure they were using the same qwen quants for every run or even test fable in their harness probably that is the least of the issues here.