I do think these “run a bigger model than will fit in VRAM” projects are necessary steps, but are they functionally useful or helpful to anyone currently? For example, is anyone out there running a big Qwen for coding on a 16-32GB machine with these techniques?

Ahaha thank you, I naively assumed the unlabeled graph in the readme was tps, not spt!

I wonder what this measures in J/token.

How many is that in tokens per Scaramucci?

that's 0.003 tokens/second. To get an hour's work done that's normally 30 tokens/second (108k output tokens in an hour) will take 416 days at this rate. And if you're using 100 watts, during that time you will spend $124.61 in electricity, as well as not being able to use your device for something else, plus the noise and heat from your device.

For $124, on Moonshot's official Kimi K3 API rates ($0.30 per 1M cached input, $3 per 1M fresh input, $15 per 1M fresh output), you can purchase 42 million fresh-input tokens, or 8.3 million generated output tokens, in whatever mix you want.

So what you get is 80x more expensive and you wait 416 days to get it.

Hah I was looking for it and couldn't work out how many years/token. 292s is pretty good.