Think about it this way: assume you supply all the housing to all the rich people. Then there still remains untapped demand of others that can be fulfilled by further production of homes for those specific people.
This story fails when land becomes restricted, which is exactly what zoning laws cause. Zoning is a big harm to the poor.
This disregards basic geometry. Sure, in some rare situations you only have one small plot of land surrounded by existing construction or natural boundaries. But, in the majority of cases, you have one large plot of land, and you can either construct one big house on it, 5 smaller houses, 10 small houses, or 200 apartments in a block. The rich are absolutely competing for this lot with the poor.
And as inequality goes up, the rich can even start contemplating buying up surrounding properties, tearing down construction, and transforming a small plot into a much larger one.
As with everything the regulator needs to strike a balance to make the market work.
The reality of zoning laws in Western countries is to provide a target for regulatory capture by the NIMBY crowd. With the result that we're systemically underbuilding housing, then wonder why we wound up with homelessness.