The beneficiaries of discrimination are usually split among class lines. So you have economically poor white folks who are indeed harmed by racial discrimination—though not nearly as harmed as the discriminated groups them selves—and rich white folks who are the only ones making money off of it.

The harm is often second factor such as the abundance of cheap (or free) labor yields less bargaining power and you end up working for less than you otherwise would have (but also the psychological harm of living in an unfair society). But next to the harm caused to those who are indeed discriminated against, the harm is rather minute.