Where LLMs as a genie are being forced out of the bottle early, the size and scale of it can like only be met by the same tools, better used and equipped and directed by humans vs the LLM iterating until it seems to work.
Centaur vs. Reverse-Centaur
Bostrom touches on this somewhat pessimistically:
> The apparent safety of a tool-AI, meanwhile, may be illusory. In order for tools to be versatile enough to substitute for superintelligent agents, they may need to deploy extremely powerful internal search and planning processes. Agent-like behaviors may arise from such processes as an unplanned consequence.