> The logical extrapolation of AI capabilities stress-testing human software is all bugs being discoverable. Hence, you have to aim for actually bug-free software.
> The only way to do that is to make your code provable. That again is only realistically doable using AI.
If you're code is "provable" doesn't that only mean you've proved the code meets its specification? The specification can still have bugs.
Yes, that's where the "engineering" part comes in.
Engineering builds models that approximate the real world under practical considerations. You e.g. have to weigh costs vs risks and apply good judgement.
In essence, you try to achieve the best possible with the resources you've got.
You are describing the difference between Verification (Does this do what we explicitly said it would do?) and Validation (Does this do what we wanted to achieve?). One is about compliance of design promises, and the other covers the issue that the various levels of design documents might have failed to completely capture intent.