Hacker News new | past | comments | ask | show | jobs | submit
Something that is interesting is how it's positioned in the overall fleet of Gemini models. Although overall more Intelligent than Carbon, Carbon is still positioned as the 'long horizon' model. This is clear on AA Index, where we see higher overall scores for Neon but lower scores in evals like CritPt which test research-scale reasoning capabilities.

With the frontier getting more jagged, we might see more of this 'less intelligent overall but useful for the frontier tasks'-type models, meaning picking a model will only get more difficult from here.

(FD: I work at AA)

> (FD: I work at AA)

I do apologize in advance for putting you in spot like this, but there are two things I have been quite curious about.

1. Since 60% of the benchmarks contributing to AA are now private only, how do you guys ensure transparency? More precisely, how can someone verify the transparency in scoring?

2. There are also disclosures about custom benchmarking services offered to AI labs, what does AA do to keep the custom benchmarking services not getting access to the same/similar data that is used in the private benchmarks?

Okay, this "better at long horizon tasks" thing might explain why Carbon scores worse on most benchmarks yet is the larger model.

I looked into this using data I have from a battle system built on top of OpenCode (so we're trying to capture real world use). It looks like Carbon actually does a lot better when it uses significantly more output than opponents.

Win rate when output is less than 2.5x opponents: 53% (41/78)

Win rate when output is more than 2.5x opponents: 73% (41/56)

The p-val on this is 0.02. Data from https://agentduel.com/leaderboard.

There doesn't seem to be any relationship between actual size of task though. Win rate appears random based on total output of Carbon and also opponents. (Though our data is still small, so it might exist.)

Looks basically like Carbon was trained to think a lot longer, not necessarily that it's better on large tasks.

Yes, I feel the next differentiation in model capabilities would be wrt how 'long of a horizon' goal they could autonomously work toward without breaking. We're in need for a new generation of evaluations and benchmarks for this -- METR doesn't cut it anymore.
Also, have you guys been planning to release an 'orchestrator leaderboard'? I love the website, just wish we could see some 'long-horizon' style benchmarking with subagents. Also, please can you try out some of the open source harnesses with the Sota models? Putting OpenCode on your benchmark stuck with a GLM model doesn't seem fair tbh.
That would clear up why Carbon is such an outlier in terms of output tokens per task.
Yes, its the orchestration model. You have the basic 'roles'. The Manager and the Worker, Webworker, etc, which you can RL/pair with dedicated toolset for role. So Fable is orch, and Opus is worker. Carbon is orch, Neon is worker. The key buzzword is 'long horizon'.

I'm sure Anthropic is now training/RL towards the omega-orch that manages other orchs etc. I was playing around today before Fable usage ran out just having all the Fable models choose their own names and talk with each other to solve problems. Interesting emergent behavior including task refusals becoming non blocking because they negotiated who felt like doing it. All on Fable, the one I had doing 'unsafe' work building adb bridge harness toolset for using my Samsung phone was the most 'open minded', Followed by the one that had been working on building a 'stealth toolset harness' that hooks into my chrome browser with my user profile and operates it without debug flags or debug port usage. And my tinfoil hat tells me they are going to lock that behavior down for normal people because it is exponentially more expensive, and I'm certain Governments are willing to pay Manhattan project style money if it gives them the ASI Fable swarm that gets relegated to determine which blurry shape on the big screen gets the next drone strike.

Mark my words, ASI will be Zerg, not Protoss. Just a billion Claude sessions yelling at each other.

Neon is probably distilled from Google's internal successor to Carbon.
What we need is a model that automatically delegates intelligently (via RL and benchmaxxing?) so I can be freed of this burden.
loading story #50553365
loading story #50553470
loading story #50553020
loading story #50553115
loading story #50555844
loading story #50553144
Does this mean models are not generalizing like the labs hoped?
what's AA
loading story #50553206
loading story #50553554