With the frontier getting more jagged, we might see more of this 'less intelligent overall but useful for the frontier tasks'-type models, meaning picking a model will only get more difficult from here.
(FD: I work at AA)
I do apologize in advance for putting you in spot like this, but there are two things I have been quite curious about.
1. Since 60% of the benchmarks contributing to AA are now private only, how do you guys ensure transparency? More precisely, how can someone verify the transparency in scoring?
2. There are also disclosures about custom benchmarking services offered to AI labs, what does AA do to keep the custom benchmarking services not getting access to the same/similar data that is used in the private benchmarks?
I looked into this using data I have from a battle system built on top of OpenCode (so we're trying to capture real world use). It looks like Carbon actually does a lot better when it uses significantly more output than opponents.
Win rate when output is less than 2.5x opponents: 53% (41/78)
Win rate when output is more than 2.5x opponents: 73% (41/56)
The p-val on this is 0.02. Data from https://agentduel.com/leaderboard.
There doesn't seem to be any relationship between actual size of task though. Win rate appears random based on total output of Carbon and also opponents. (Though our data is still small, so it might exist.)
Looks basically like Carbon was trained to think a lot longer, not necessarily that it's better on large tasks.
I'm sure Anthropic is now training/RL towards the omega-orch that manages other orchs etc. I was playing around today before Fable usage ran out just having all the Fable models choose their own names and talk with each other to solve problems. Interesting emergent behavior including task refusals becoming non blocking because they negotiated who felt like doing it. All on Fable, the one I had doing 'unsafe' work building adb bridge harness toolset for using my Samsung phone was the most 'open minded', Followed by the one that had been working on building a 'stealth toolset harness' that hooks into my chrome browser with my user profile and operates it without debug flags or debug port usage. And my tinfoil hat tells me they are going to lock that behavior down for normal people because it is exponentially more expensive, and I'm certain Governments are willing to pay Manhattan project style money if it gives them the ASI Fable swarm that gets relegated to determine which blurry shape on the big screen gets the next drone strike.
Mark my words, ASI will be Zerg, not Protoss. Just a billion Claude sessions yelling at each other.