I have not drilled down into the data, but I think it highly likely poor people did not benefit.

I have read that the Empire was a fiscal net negative. That was offset by it being a free market area with policies that favoured the UK, the benefits from that went primarily to a small minority.