Bro 500 Aggregate. so that's 500 * 60 * 60 = 1.8M output which is .5$ at best... Not including pre-fill and stuff.

This is not the real margins, even if you are selling to 8 users it's 90 tps per median stream. So assuming that .6-.7$

This is not even remotely worth it.

You need to 3x this tps(~1500 tps) to be worth it, and that's what most providers are doing, at 20-30 users at 50-60 tps with better optimized batch processing and kernels you can make some profit.

Agentic workloads are somewhere around 1%/0.5%/98.5% input/output/cached tokens. Cached tokens are pretty much free for inference providers (if they implement sparse and compressed attention properly) and throughput for input tokens is much higher.

Lets assume that you've got 2 million input tokens, 1 million output tokens and 98.5 million cached tokens to process. That would cost 2 * $0.14 + 1 * $0.28 + 98.5 * $0.0028 = $0.8358 with DeepSeek API pricing.

For comparison, it would take 2M / 8000 + 1M / 800 = 1500 seconds to process this amount of tokens with the linked framework, which is about $0.83 when we assume $2/hr for one MI300X.

However, other inference providers have 10 times higher prices for cached tokens, which results in a comfortable margin.

And we should not discount that DeepSeek also gets paid in data, which is probably more valuable to them.

And I believe that this framework still has some room for optimization for generation with high batch sizes.

> They are selling at a loss?

Definitely not. Inference is not as expensive to operate as many people seem to assume. The frontier labs are probably making a lot of money from selling tokens. It’s covering all of the R&D costs like salaries, collecting training material, and running the large training operations that costs a lot of money.

They claim their advantage is knowing how to serve their models efficiently, which is quite possible since they design for it.

They get all our invaluable data which they'll use to train the next model, to get more data, to train the model after.

I think Deepseek is selling roughly at cost (perhaps a slight premium). They don’t guarantee that they don’t train on the submitted prompts, so I suspect they are mining the data. Mining for what? Well, who knows. Best case, mining to make Deepseek better. That said, I use Deepseek all the time. It has done a whole lot of ‘ls’ commands on my system, though.

Use nvidia hardware instead and use a larger cluster serving many more users concurrently. Easily 10x–20x higher token rate per GPU with public solutions like dynamo and sglang.

If you don't do any attention steering, custom decoding or meddle with the weights maybe. Services are worthless unless all you do is write positive prompts.

As others have mentioned, there's the privacy factor as well.

> trying to run it locally or with your own hardware is pointless.

Serving local models has advantages other than price. If you work in restricted industries, or have a strong need to protect your IP, or if you just value privacy more than cost, you now have options.

With the cost of electricity, hardware depreciation and tok/s it rarely makes sense to run locally.

I don't think inference is subsidized, it's the training. So what happens is, there's no new models anymore or are released slower.

Well, if no body can say when it will pop, then can we really say it's a bubble and it's overvalued?

I can tell you it will pop in 10 years and when it pops, it will still be 20x bigger than in 2026. Does that even make any sense?

People said AI bubble will pop soon in 2024 and that it was overvalued. Turns out, many AI stocks 10x, 20x since 2024. Actual usage has gone exponential as well. Anthropic revenue went from $100m ARR at start of 2024 to $80b ARR today.