1,5k. For two months of that spend you could buy a machine that can self-host decent models, plus a year's worth of electricity. It's not up there in terms of quality, but with a bit more effort it works pretty decently. I'm completely baffled that that's not way more common, is it really just the quality?
Second here. From recent Alibaba Qwen conference: the all-in-one box (DC in a box - I think I was called Apsara, 0.6x0.6x1.5m) plug and play, 1.5TB GPU RAM, capability to run in a fully air gapped environment, any open models... All of that is roughly $300k one time.
And this box can do non LLM tasks as well.
Performance (throughput) around 20k t/s. Delivery time - around 2 months.
For any medium sized company its perhaps cheaper to just buy it once than spending 1.5k for cloud per user
loading story #48401472
Decent vs best-money-can-buy. Further, a self-hosted LLM will be much slower.
loading story #48397353
I'd think for most companies the pace of change is too high at the moment. Give it a few years, a bit of a plateau in the improvements in frontier models and I can't see how many of these companies don't implode under the weight of competition on inference prices.