Hacker News new | past | comments | ask | show | jobs | submit
Can you share more, or direct us to more resources around local setups? For starters what's the upfront cost, what are the space, weight, energy requirements, etc.
Really depends on what GPUs you go with but I hate proprietary software so I went with AMD AI PRO r9700 cards which you can still pick up used on ebay for $1500 used on auctions. Started with 1 then expanded over time as I won auctions. Though on a lower budget my $400 16gb 6800 XT can run Qwen 3.8 27b which I could absolutely get by with at significantly reduced speed.

My entire 12u rack has roughly the same capability as an H200, though at maybe 15k all in with mostly used parts vs 50k for a used H200 checking todays ebay prices and that is -just- the card you still need to build a system for.

Consumer cards are a lot more power hungry than enterprise cards and they take up more space. I run my setup next to my dryer, running on 240v for 15% better power efficiency.

Also using transparent memory encryption on a Xeon and an EPYC for confidential and private inference guarantees no third party can offer me. Of course I can run abliterated models that do any security research I want without refusals.

Strata and Radiance are the state of the art ATM for running big MoE models on small consumer GPUs at high performance.

loading story #50564246
I would laugh heartily if this user turned out to be a Black Hat who just wants to gush about his gear, because this is exactly the rig I would deploy if that was my aim.
As a career security engineer that has found and set 0days on fire that could have made me tens of millions of dollars many times, I am very curious what point you are trying to make.
loading story #50568415
loading story #50563313