Working on a blog post to document it all, but that will take a few more days. The short version: the big rig is probably $25K, ttft is dependent on how much prefill you have but typically a couple of seconds (2K prefill/sec in one setup, about 500 in the other), tgs 100 to 150 or so depending on what it is doing and how well the speculative decoder works with the current context. Electricity is pretty bad, full tilt it's about 2500W or so but I've power limited it to 1500 to keep it civil. It is not cheaper than using a frontier lab (or at least, I don't think it is), but the independence bit is the important part for me and I don't care if that carries a surcharge, even if it is a substantial one.

I mean, I just have 4xV100S GPU and a Macbook Pro w/48GB of RAM and I find it kind of amazing.

My server can run Qwen Flash Next at 500 t/s aggregate (about 200 t/s single user). It's pretty great. I use it for local coding and all my worker agents. It is about as reliable as Sonnet 5 for agentic coding. I make 3D models, keep the home network and server updated, fully built ansible files.

You do NOT need 20K in hardware to run these models reliably. I built my own inference engine from scratch for fun. You can run Strata on 16-32GB cards at very respectable token rates if you have the standard RAM to go with it. I strongly encourage people to go use this model on OpenRouter and try it out for tasks.

I also have a somewhat lobotimized quant of GLM Flash 5.3 working and it's pretty decent too.