I think this misses something.

AI is going to struggle at building a consistent internal model of the domain into the software unless you’re able to give a structured explanation of the domain.

If you’re just giving it a set of inputs and expected outputs, it’s not going to generalise well and fail at out of sample input, unless the AI already understands the domain from its training set.

Being able to give a structured explanation of a domain (and being able to judge if the internal model of the software makes sense) is not the same as having experience in a domain.

Lots of ppl with domain experience can tell a right output from a false one, but can’t tell you why.