If you are going down this path, MI350P and RTX PRO 6000 Blackwell is what you would want. A lot more compute a lot more vram bandwidth.
> A lot more compute a lot more vram bandwidth.
But this gives you significantly more memory capacity for a compute/vram bandwidth tradeoff (i.e. arithmetic intensity) that's ultimately comparable to Blackwell, perhaps even more skewed towards compute. This seems good if you care about being able to store larger sparse models and more KV caches because that ultimately bottlenecks your ability to do inference.
Maybe. I don't know power for the MI350, but the RTX PRO 6000 is normally a 200-300W card, so you pay 2x the power for 3/4th the RAM and 3x the bandwidth. The performance frontier looks very different in terms of power to hold a mid-size model and throughout per user or across users. Price and inference stack would be big factors in what I would choose.
MI350P, H200 NVL and Nvidia RTX PRO 6000 are all 600W but compute wise all of them should be 3 to 4 times more performant than A100 Ultra according to their specifications.
https://www.servethehome.com/the-amd-instinct-mi350p-is-a-hb...
600w is way off the vf curve though... Tiny bit of tuning and you'll get ~80-90% of the performance at ~150w nominal. At least that's what I got out of some pro 6000s when I spent 15 mins to underclock it.