Either:
1) You also don't tolerate the below median earner who is native to your country
or
2) Your tolerance is dependent on citizenship not just income
If you're #1, that's a problem for your fellow citizens whom you don't tolerate.
If you're #2, you're telling me to not bring my higher earning skills to your economy.
Doesn't matter if you didn't mean it that way, you still won't get me spending the money I earn on your economy so you won't get rich from me.
Why can't I want to minimize the number of unskilled outsiders (with different values, etc.) because they may cost more while overlooking that fact for those with obvious economic power regardless of where they are from.
I know it hurts to hear: people with wealth are desirable guests and citizens.
A country's citizenry is much like children: some are going to be shitty, but we still support that limited group because of arbitrary moral obligation (perhaps inspired by the fact that we want our "own" to continue.) We're not obligated to extend this tradition to anyone else for any reason.
> you still won't get me spending the money I earn on your economy so you won't get rich from me.
Thankfully there are billions of people in the world and they're literally dying to get into the US. H-1Bs quotas are filled every year - there's no shortage of high-average earners wanting to come here, either.