Exactly - when things are happening "naturally" for some value of "not artificially constrained" you find that people move over time and what were the luxury dwellings of 20/30/50 years ago are the new "starter homes" of today.

When supply is artificially constrained, the old homes get torn down and replaced with luxurious ones - without increasing dwelling spaces available.