Hacker News new | past | comments | ask | show | jobs | submit
I'm running qwen3.8-flash-next at 4-bit quantization on two AMD Radeon AI PRO R9700 cards at 75 tok/s. And I can actually run two models in parallel at around 50 tok/s, so you can get by with only one card. I switched from NVidia to AMD for a fully open-source stack for my local inference.

AMD Radeon AI PRO R9700 is $1600 right now (I bought them last year for $1300).

They are also quite power-efficient. My setup typically uses around ~300W during inference.