I think someone who likes AI thinks it’s good, someone who doesn’t, doesn’t think it’s anything impressive, and concretely nothing of any significance has happened to the world, certainly nothing like the crowdstrike shutdown so I don’t find the capabilities to be anything impressive.
I would hypothesize that all the long-held "easily discovered if you look in the right place, but never reported and so never fixed" 0days in operating systems that criminal gangs and militaries have been relying on (to do cyberattacks with, unlock devices for forensic data recovery with, etc.), are increasingly being exposed and cleaned up, much faster than new such exploits can be discovered; and that this may have given an edge to whichever side of these conflicts don't rely so heavily on hacking the other to achieve their goals.
The 9 bugs I got were clear security bugs under the Fil-C definition of memory safety. One reason why my experience could be an outlier is the effort I’ve gone to defining what the security properties of Fil-C are, to the point that there’s a crystal clear falsifiable boundary. So the bugs aren’t a matter of opinion or judgement. That helps clankers and it helps me when screening the bugs.
> concretely nothing of any significance has happened
This is a good point, but “cyber apocalypse dooming is an exaggeration” is a different statement than “bugs found by Anthropic aren’t real”. I agree that the dooming is an exaggeration. I don’t agree that Anthropic’s bugs suck.
Two reasons why bugs-found-by AI-are-really-scary and nothing-is-happening can be true at the same time:
1. Anthropic is sharing the bugs found by their most capable model with defenders, not attackers. Defenders are working overtime fixing those bugs before the embargo runs out.
2. The attackers that have access to Anthropic-class capability are using their bugs surgically. They aren’t using them so widely that folks would notice. That strategy is business as usual for advanced attackers
For example it's definitely not OK that you can impl Pattern in a way that causes non-UTF8 data in a string with safe code. But you probably didn't let random attackers impl Pattern, on a nightly Rust where that's even possible, and so that's not going to cause problems in our real world - even though it's wrong and must be fixed.
A model with all the world's knowledge might know exactly how to weave them together into a successful exploit. And if tomorrow's models are even more capable, perhaps what we're experiencing now is just a fire drill for what's coming.
But they were all things that unambiguously violated the Fil-C promise
Building software boils down to defining things correctly. So, it sounds like you did most of the work here. Imagine that you took your definition and moved it into a formal specification (programming language) and then ran that to verify the security properties of your system. What's different?
Or, imagine that you took your original code as your specification language and then iterated on your understanding of it until it was correct. What's different?
Attention highlights and resolves bugs, so that's the baseline. In order for AI to get value, it has to be some kind of multiplier on your attention, but the reality is generally the opposite.
I wonder what it's like maintaining a project where you regularly get competent contributions. No wonder they are less impressed with AI - they are the 0.1% of maintainers who had something better.