Hacker News new | past | comments | ask | show | jobs | submit
Flavius | on: GLM-5.3: Frontier coding with emergent cyber capabilities
> run free LLMs locally at native speed

This reads like a hallucination. What does native speed even mean?

kyxsc | parent | next
for example, models running at like 100-150 tokens/second (or faster!) vs 15 t/s

(fable/sol are ~60 t/s, and OpenAI just announced their Cerebras partnership(?) for "ultrafast" mode of 750 t/s)

models aren't able to run that fast right now on our consumer/prosumer hardware. M5 Max for example has a memory bandwidth of 600 GB/s. a 5090 has 3x that, so running the same model on a 5090 is that much faster (provided the model is within 30GB).

running a bigger model on an M5 Ultra is still much slower than running it on a Blackwell chip with sufficient vram, CUDA being a major difference. if apple can bridge this gap, interesting things will happen... and just imagine if M7 Ultra has comparable speeds to Blackwell (or even Rubin)!

toasty228 | root | parent
Meanwhile the GB300 used by hosted llms:

GPU Memory Bandwidth: 7.1 TB/s Interconnect Bandwidth: 900 GB/s bidirectional

https://pi3g.com/nvidia-gb300-specifications-including-memor...

If you think M7 will hit even 15% of these speeds you're very optimistic.

andsoitis | root | parent
A hosted instance serves multiple customers at a time. A local model only one.
toasty228 | root | parent
How many though? At 1m context you quickly fill a full gb300's 280gb of memory
lmpdev | parent | next
I assume they mean same t/sec as a SOTA cloud model
loading story #49296809