This feels like just another variation of “it’s not the tool it’s the user,” which is too frequently leveraged during any discussion about AI. It makes it impossible to honestly assess the quality of the tool. You can always just say “people are doing it wrong.”
Idk, I think it pretty clear to me that
“Find the issues in this code” vs “write this feature” are different problems.
The first one is easier because the feedback loop is llm to code/test and trivially scalable.
The second one is llm to human which is going to be slower because it can’t read your mind.
The tighter and more deterministic the loop the more llms excel.
its certainly benchmarkable.
make a bunch of "build the feature" vs "heres a detailed spec" prompts, and see which one does better.
if you take out the AI boogieman, and instead go with "next token prediction" then "people are doing it wrong" is much more reasonable
I mean that's what you would expect from a new and rapidly-evolving tool. Being able to identify the kinds of work where people typically use it sub-optimally is a useful thing though!