It seems to me that the market will select for urban sprawl though which is a negative for society but has the highest margin. E.g. Houston suburbs, miles and miles of cheap to fab single family homes that turn it into a suburban hell scape where you have to drive everywhere.
I don't think the free market is giving the promises you say it is - supply isn't elastic for real estate if nobody's building because there's no margins. Demand can be anywhere really.
I like to look to Tokyo for an example. Small lots, extremely predictable regulations (that are still strict enough to ensure a safe living situation), fast approvals, mean it's much faster and easier to throw up an 8-10 story apartment than say downtown Austin, and so even today they keep doing it despite land in Tokyo being very expensive. And, no sprawl.
They can throw up 8-10 story apartments in Tokyo despite land being very expensive because they tear them down and rebuild them after 20-30 years. Also, Tokyo outside of a few areas isn't that tall, it is definitely dense, but 3-4 story tall building dense (those homes are also torn down and built anew ever 20-30 years, so construction buzzes in Tokyo).
It would be better if you considered new actual living capacity in Tokyo rather than just new constructions.