Yes, you can also easily prove the wrong thing.
That's why it doesn't suffice for only LLMs to know the relevant stuff.
You can't have a zoo run by animals.
Aren't all zoos technically run by animals? :D.
For months after some researchers “proved” our code signing system sound I was still finding security holes in the system. To this day I’m not sure what they proved. That the architecture worked? We’d already proven that to a hundred people.
Correct, because when there was no LLMs all code was perfect and bug free.