Its a possible as an attack vector for sure.
But in comparision to nature, compute can speedrun all of this with RL.
This has different failure modes than monocultures, which would also be bad for us: a collection like that behaves as a species, and can evolve faster than one single thing, and evolution is a blind watchmaker that cares only for success rather than details.
Even a lot of what software professionals do isn't verifiable.
For a human you need to train everyone every year and if you have more and more material to teach, harder it will be for the humans.
Not so for AI.
Then we will also see more simulations, an AI will be able to play through scenarios and we will have models being able to invent new scenarios.
While this is happening, hardware will be faster, more and more improvements on every front. Humans have not progressed really
More generally, if you assume that magic can replaces the laws of statistical learning, then yes, AI can do everything and we're all doomed.
I reject this premise on the basis of knowing how RL and LLM training works.
How many people learn one particular job? If its a job which educates 10.000 people per year, when is the financial motivation big enough to create enough training data to make them obsolete and transfer the learning into an AI model?
Also we don't know yet were the journey goes, 'dreaming' models finetuning over night might help.
Or its already just enough to have the biggest RL Loop in history of humanity: everyone of us using the chatbots and giving them signals like thumbs up and thumbs down and stuff like "DUDE WTF I TOLD YOU..." you know :)
Then we also know that the frontierlabs are paying experts and people with PhDs to generate challanges and answers.
Educating just a 1000 people every year over and over is very very expensive even if LLMs need more training data. In worst case frontierlabs will just keep investing in generating traingindata manually with their earnings and it takes. longer.
Or we find a breakthrough like how LeCun is complaining all the time that LLMs are not learning properly. We are still early and progress is still happening fast.
Or maybe cut back on the acid.
Thinking it can detect the unknowable at the time is a big ‘ML magic’ thinking error.
It’s the same thing that fed the grey goo panic around the 2000’s.
Turns out, entropy and the law of thermodynamics applies to nano machines too.
With AI you have to teach one model once. It makes a mistake? Teach it and it will not make it again.
This is a lot cheaper than training every pilot on the planet and its a lot more persistent then training every new pilot
Has this been your experience? Models still invent answers for me. Directives go unrespected, etc.
I expect it to get significantly better in the next few years but I also expect either to see continues improvement or new optimized models for specific tasks.