That assumes immigrants are average people, but they are not they mostly work in some sectors. Those sectors will see a wage dump, other sectors might see a wage hike to compensate though.

For example if immigrants are mostly highly paid programmers, you can expect waitresses etc to get a wage hike, but if immigrants are mostly uneducated young women then waitresses will probably see reduces wages.

If you look you can see the groups who compete with the immigrants tend to be more hostile towards immigration, while the groups who doesn't see immigration in their sector aren't as hostile. Most immigrants tend to be men for example, so we would expect men to be more anti immigration since their jobs see more competition from it, and that is also what we see in opinion polling.