All of them were real bugs. None of them were false positives.
The quality of the reports was exceptional. (My baseline is doing security defense on a browser engine, so my baseline has a large sample count and is probably representative of what experts are capable of.)
The quality of the proposed fixes was alright. I took some of the fixes verbatim. I used most of the fixes as significant inspiration for doing it my way.
The quality of the test cases was exceptional.
Maybe I’m special somehow! But I just want to call out the enormous chasm between what I saw of anthropic’s capability and what TFA says
If you told me that some significant number of Ant findings were duds, then I could reconcile that with what I saw first hand.
But TFA is making a more extreme statement
> The slop is gone
> A sizeable portion of users seem to still believe we still suffer from a lot of made up and hallucinated reports by these AIs. We really don’t.
> Almost all the stupidity and lies we saw in the vulnerability reports made by AIs in the past have ceased. Since spring 2026 the slop reports are practically gone. Our current challenge is much harder: high volume, high quality.
I think someone who likes AI thinks it’s good, someone who doesn’t, doesn’t think it’s anything impressive, and concretely nothing of any significance has happened to the world, certainly nothing like the crowdstrike shutdown so I don’t find the capabilities to be anything impressive.
I would hypothesize that all the long-held "easily discovered if you look in the right place, but never reported and so never fixed" 0days in operating systems that criminal gangs and militaries have been relying on (to do cyberattacks with, unlock devices for forensic data recovery with, etc.), are increasingly being exposed and cleaned up, much faster than new such exploits can be discovered; and that this may have given an edge to whichever side of these conflicts don't rely so heavily on hacking the other to achieve their goals.
The 9 bugs I got were clear security bugs under the Fil-C definition of memory safety. One reason why my experience could be an outlier is the effort I’ve gone to defining what the security properties of Fil-C are, to the point that there’s a crystal clear falsifiable boundary. So the bugs aren’t a matter of opinion or judgement. That helps clankers and it helps me when screening the bugs.
> concretely nothing of any significance has happened
This is a good point, but “cyber apocalypse dooming is an exaggeration” is a different statement than “bugs found by Anthropic aren’t real”. I agree that the dooming is an exaggeration. I don’t agree that Anthropic’s bugs suck.
Two reasons why bugs-found-by AI-are-really-scary and nothing-is-happening can be true at the same time:
1. Anthropic is sharing the bugs found by their most capable model with defenders, not attackers. Defenders are working overtime fixing those bugs before the embargo runs out.
2. The attackers that have access to Anthropic-class capability are using their bugs surgically. They aren’t using them so widely that folks would notice. That strategy is business as usual for advanced attackers
For example it's definitely not OK that you can impl Pattern in a way that causes non-UTF8 data in a string with safe code. But you probably didn't let random attackers impl Pattern, on a nightly Rust where that's even possible, and so that's not going to cause problems in our real world - even though it's wrong and must be fixed.
A model with all the world's knowledge might know exactly how to weave them together into a successful exploit. And if tomorrow's models are even more capable, perhaps what we're experiencing now is just a fire drill for what's coming.
But they were all things that unambiguously violated the Fil-C promise
Building software boils down to defining things correctly. So, it sounds like you did most of the work here. Imagine that you took your definition and moved it into a formal specification (programming language) and then ran that to verify the security properties of your system. What's different?
Or, imagine that you took your original code as your specification language and then iterated on your understanding of it until it was correct. What's different?
Attention highlights and resolves bugs, so that's the baseline. In order for AI to get value, it has to be some kind of multiplier on your attention, but the reality is generally the opposite.
I wonder what it's like maintaining a project where you regularly get competent contributions. No wonder they are less impressed with AI - they are the 0.1% of maintainers who had something better.
OP's post adopts a deliberately skeptical angle, it is a blog post for a company that advertises the ability to spot and patch vulnerabilities. They have an intrinsic motivation to minimize the believed effectiveness of wholesale SOTA-model cyber superiority which erodes their moat.
Im about ready to stop using them overnight in this autonomous fashion. Its a great "intelligent" fuzzer but pretty mediocre when its steering the ship by itself.
By contrast, when <tool> announces they've found XXXX vulnerabilities in Linux, it makes the news literally everywhere. Even if 0% of them turn out to be real, it doesn't matter - they've already gotten what they want out of it which is the round of coverage about how incredible <tool> is and its the end of the world