I should probably have said the change in life expectancy is a weighted average, weighted by how far you are from the average. If average life expectancy is 80, removing a data point where somebody died at 40 has 8x the effect of removing a data point where somebody died at 75.
In case anyone else is curious about the specific term for the concept you are describing, it's https://en.wikipedia.org/wiki/Leverage_(statistics)
(To reproduce exactly the scenario being discussed, you fit a constant-only model to the data using least squares: that gives the average as the best fit. Then, you measure the leverage of each point of interest.)
Ok, that makes a lot more sense in light of your argument!