In case anyone else is curious about the specific term for the concept you are describing, it's https://en.wikipedia.org/wiki/Leverage_(statistics)

(To reproduce exactly the scenario being discussed, you fit a constant-only model to the data using least squares: that gives the average as the best fit. Then, you measure the leverage of each point of interest.)