Hacker News new | past | comments | ask | show | jobs | submit
What makes that kinda complicated is that multi-user throughput of LLMs scale well but single-user performance often stays constant at low ends. If you could saturate e.g. 16 concurrent session-month of demand, you can just go buy 16 of 32GB GPUs and start charging monthly for inference. That could work if you had e.g. over thousand total employees with hundreds of devs eager to trying it out, but only if the company is also interested in a private inference experiment.
You're talking about multi-session vs. single-session throughput. A single user can easily leverage multiple sessions via e.g. subagent swarms, especially on a lower-end setup where any single session is going to be quite slow. Saturating utilization during off-hours is harder but potentially quite feasible by assigning lower priority, unattended tasks/inference loops.