Hacker News new | past | comments | ask | show | jobs | submit
If your benchmark puts DS4.1-Flash at the same tier as K3 or K3.1, I'm sorry to say your benchmark is broken. It really isn't, as much as I wish it was.
We allow early voting. This is against what most arenas do, but is on purpose. If the model is faster to do simple stuff, it should be chosen. In this sense, DS4.1-Flash is better on many tasks.

You can remove early voting from leaderboard and they are no longer the same.

At the point where I choose an expensive model like Kimi 3.1, speed doesn't matter, at all.

Usually, it's because all else failed, and I have to resort to burning Tokens with more expensive/better models, so a few minutes here and there really should not matter.

I think you’re alone to think time for task completion doesn’t matter
loading story #50376873
But how do you verify a fair or representative distribution of complexity?
We validate people are doing real work using LLM judges.
I tried both of them for agentic coding and didn’t feel like K3 was adding so much benefit to justify its almost 10x price (on the modem provider I’m on).

In fact, I barely see a difference between the results at all. The models being tied smells correct to me.

I didn’t try 3.1 yet, though.