Hacker News new | past | comments | ask | show | jobs | submit
tmpz22 | on: Claude Fable 5.2 and Mythos 5.2
It would be trivial for them to opt-out accounts that are doing benchmarking from a planned regression.

I really really really do not trust companies on the verge of IPOing from goosing their numbers. Even if the lawyers are up their ass, the social pressures are more than enough to compel the behavior. For every good person willing to do the right thing even at the cost of a delayed career progression, there are 20 people willing to do anything not even for a dollar but for an "atta boy!" from the boss.

Wowfunhappy | parent | next
If it's a private benchmark, how would they know it's a benchmark?
krackers | root | parent | next
Look up eval awareness. There are all sorts of subtle signals an LLM could use to know if something is shaped like a benchmark or not. But that's not a bad thing, every animal behaves differently in play vs real-life.
Wowfunhappy | root | parent
Yes the model can do that, but it would not be "trivial" (as GGP had said) for Anthropic to opt out accounts performing private benchmarks from this theoretical model downgrade people think is happening.

The most plausible approach would be for Anthropic to train some separate classifiers to detect benchmarks. I'm not saying it's completely impossible but IMO this is conspiracy theory levels of effort.

tmpz22 | root | parent
In the context of public relations they would not need to ID every benchmark only the most credible or most public benchmarks many of which are openly available on github and whose problem-set could be fingerprinted.

You're absolutely right that it is a game of whack-a-mole and eventually we may have leading private + credible + popular benchmarks that ultimately dissuade the behavior, if it can be dissuaded.

cft | parent
That's a major reason to use local models. Or open weights models on a trusted cloud provider. Even if they are way less capable, they're more predictable