My experience with AI-code is that it's extremely defensive and good at minimizing 'obvious' mistakes.
Are we comparing to old models generating code? Or mythos-class models?
There are still folks, many of them, who think that time to market is everything, and anything beside it should be ignored. "It compiles? Ship it!" is a phrase from 1990s; LLMs just make this approach much easier. We will have plenty of machine-generated software with the security posture of a colander hit by a buckshot.
Nowadays, the frontier is pretty decent all around, but it used to be generated php/javascript was questionable, ts was a bit better, and Rust was consistently decent.
Even just setting the max width of your README to 80 chars vs using full markdown could get you very different C code.
My take is that its pretty easy to write _much_ more secure code; use fewer dependencies and the ones you have should only do what you need them to do with no superfluous unused code compiled in. Configuring that used to be too much work, now its a single prompt.
Now I wonder: given that the LLMs are steered by the companies who own them, doesn't it mean that they can somehow make security better? It cuts both ways obviously: that a company can read and generate a large part of the code in the world is a risk.
As an open source maintainer, I can most definitely tell you that the quality of the average PR is now better. The problem I am facing is more that some people tend to "flood" and misuse AI in different ways. But their AI writes better code than they do.
Tell an LLM to break into its own system as part of a feed back loop, but with no context of the initial creation? It should iteratively get to 'safe' shouldn't it (subject to the tokens you want to spend)?
No doubt because us unwashed masses aren’t blessed with the super special “security” ones.
We might be looking for exploits for bad reasons, I guess.
My vibe code friends have pretty much infinite security issues. Don't even have basic data modeling at client-server boundaries or anything. Also had Google analytics embed that read and saved everyone's credit card details as clear text in analytics.
Seems to make pretty much infinite mistakes if you don't take care and tell it what not to do or how to build certain features.
The issue then is that most people aren't actually good at keeping an eye out for security issues in code proactively, so things slip through the cracks (whether human-written or not), just at a much accelerated pace.
I'm not at all saying that "oh 3 weeks ago, discard", but three years ago absolutely has a "before and after".
Three years ago AI generated code could basically not be used at all, and I had to rewrite just about every line. Today I have to watch out for generated code making steps in the wrong strategic and architectural direction, and over engineering, no longer hallucinating nonexisting APIs and forgetting to check the auth token.
I still look for those kinds of mistakes, but I think citing a 2022 paper for "can AI do X" is not useful.
Having properly validated libs with separate concerns and a lot of eyeballs (and agent sensors) on the code so that it can be integrated cleanly in other products. Static code analysis for our apps. System prompts that require output to be secure, etc.
All of these practices that have been touted to be obsolete because agents can now just generate tailor made code just when and where you need, are anything but obsolete. On the contrary, they are paramount to the future of software engineering.
> Overall, we find that participants who had access to an AI assistant based on OpenAI's codex-davinci-002 model wrote significantly less secure code than those without access.
That model came out in March 2022, it's GPT-3.5 !!!
I'm sorry but it's absurd to use that as evidence for anything related to AI coding today.
You can't just hedge by saying "it's somewhat older". It's like discussing dunno a new train connection between cities and citing a paper measure steam locomotives.