Couldn't it also be case where the pesticides are fine to use on potatoes but not on tomatoes, for example?
No, pesticide are volatile and can poison water deep in earth no matter which thing was the initial target.
That's not really true. Most pesticides have a PHI (pre-harvest interval) where they will be broken down by microbs or sunlight before harvest. There's also pesticides that are more stable and thus can't be used on plants meant for human consumption, but even then most of those do not last very long in the soil. Microbes are very efficient at breaking down chemicals.
And then, after all of that, you still have: the solution to pollution is dilution.
What's toxic on a fruit tree for human consumption, is likely not toxic on thousands of cubic yards of dirt or millions of gallons of ground water.