This is not a historical perspective. Around 1900, it were the socialists who opposed immigration as a tool to drive down wages:

https://www.cambridge.org/core/journals/international-review...

Back then the right wing were the industrialists who wanted immigration.

Hitler blamed all sorts of people, including the socialists (who were against immigration). Using Hitler to shut down any discussion about immigration is not very productive. Obviously there are limits. Less so in the U.S. because there is more space, but in Europe everyone is already living in tiny overpriced apartments on top of each other.