How much effort do you put into prompting it? Are you asking it one shot with a simple prompt? Are you giving it to multiple agents and giving them a workspace to collaborate in? You might give it to a trusted colleague who does not know the solution along with an agreement to jointly publish, ask them to go through the process of solving it with agentic support, and then compare your solutions.
Edit: to be clear, I completely believe your experience - just pointing out that it might be interesting for you to actually “try” to solve it thoroughly with the tools, though it is obviously hard to do that in a fair way when you know the solution.
Enough! I think it's a reasonably fair test of the models. Let's put it this way, I use the same process to resolve problems outside my peak expertise, and I find it works well.
I think the solution is easy enough that any competent colleague would come up with it. Some variant on it has probably even been published (hence why I'm not chomping at the bit to publish). I think it comes down to the models being extremely spiky, and the problems they're trying to solve being easily verifiable in code.
loading story #50253656