Not just billionaires so much as multi-billion-dollar corporations that are too big not to be stronger than (indebted) government and control politicians.
It's been a while but lots of the real gems (precious metals too) have already been sold for a profit so there's not as much upside as there was traditionally.
Without hard-asset inflation, the dollar could turn out to be one of their least-performing assets, and you know they can't have that.