We know the process to train a model, but when a model makes a prediction we don't know exactly "how" it predicts the way it does.

We can use the economy as an analogy. No single person really understands the whole supply chain. But we know that each person in the supply chain is trying to maximize their own profit, and that ultimately delivers goods and services to a consumer.