Frontier AI capabilities also include practical token rates. My current workload would require around 300 tokens/s continuously looking at the last 30 days of token spend. More realistic would be 1500-2000/s for human in the loop activities. I can't babysit the thing 24/7.
How far are we from this kind of throughput with local models?
Isn't this pretty much backward? If the thing spit out codes so fast, you'd have to babysit it. Slower throughput let you do something else while you wait.
Also throughput pretty much depends on compute capabilities and active weight. So if someone build a A0.1B then maybe you can look at 1500t/s of pure uselessness.