The problem is that it seems to be an A/B testing thing:

https://news.ycombinator.com/item?id=50422951

So only if you were "selected" for their experiments, your account was nerfed.

Its incredibly easy to roll out a compute instance to X% of users, or route traffic between compute instances

so not everyone will have the same experience. If I was saving on costs that would be the easiest way to do it without deploying, without any nefarious intent. The persons in charge of that likely don't even know about how individual users are impacted

"Nerf tracking" is adversarial and exploratory. It's not something Anthropic wants anyone to do, and it's not something that the "nerf tracking" designers are necessarily doing in a foolproof way. Don't assume too much from them either way.

These nerf trackers do not work. I wish people would stop posting them.

They don't detect a quantized model until the quantization gets down to 4 bits or less.

Humans absolutely do detect quantization well before 4 bits...

Try it yourself if you want. Try a small local model, both quantized and not, and I would be very surprised if it isn't immediately obvious which is which.

Nothing stopping Anthropic (or other labs) from pulling a VW Dieselgate and ensuring that benchmarks are routed to a non-nerfed model.

Should be fairly routine for anthropic to game a nerf tracker?

That must be on anthropic's benchmaxxing CI test