Concretely, I would like to see it benchmarked:
Have Opus 5.5 implement the authentication, with a 6.1 Sol reviewer. Out of 100 times, how often is the result insecure?
I wouldn't be surprised if it's zero times.
There is also the possibility of specifically prompting for a security audit on top, which would perhaps further lower the risk of overlooked issues.
Now one can say that due to the non-deterministic nature of the tools, there is always a non-zero chance of producing insecure implementation.
That's not really important here - obviously manual review isn't deterministic either, the risk will never be zero.
“Authentication” is extremely far from being the only relevant characteristic of security. That in fact is the whole problem with security: it’s a pervasive concern that has to be considered throughout a system, at every level.
> I wouldn't be surprised if it's zero times.
Statements like this concern me.
Instead I mostly read such opinions based on active imagination and no concrete experience with using AI for such work in a setup as described.