What you're missing in your calculus is that by the time the local maxima is reached, the business has either pivoted to something else or is throwing money at the problem to make it go away.

My biggest issue with AI is not really technical debt, it's engineers losing sight of the system's internal working. This can make it difficult to reason about the code, fix bugs or security issues.

I just don't think we have the tools right now for engineers to accept software becoming a black box.