Hacker News new | past | comments | ask | show | jobs | submit
aetherspawn | on: Qualcomm Cloud AI 100 Ultra – 150W AI Accelerator Card with 128GB at 548GB/s
Ok so it’s significantly worse than Apple M6 and we don’t know if it’s cheaper or not

Edit: Ah sorry you’re right, the bandwidth is lower but the TOPS is a lot higher. I’m not sure how much effect you get from higher TOPS with such low bandwidth though.

zozbot234 | parent | next
> I’m not sure how much effect you get from higher TOPS with such low bandwidth though.

Raised arithmetic intensity. It would be quite good for LLM prefill, and decode would want to adapt by pushing parallelism higher (saving on memory bandwidth by reusing the same layer weights for multiple streams) or, if unable to do so (e.g. due to KV cache constraints) by resorting to speculative decoding (where the expensive 'validate' step handles tokens in parallel within a single stream). It does not make the platform inherently unviable. It would also do well at other tasks such as image/video generation, which is already quite compute bound.

dist-epoch | parent | next
Apple M6 has 80-100 TOPS.

Qualcomm Cloud AI 100 Ultra has 800 TOPS, almost 10 times more.

grosswait | root | parent | next
This could be very fast at running embedding models (if it supports the right modes)
ipsum2 | root | parent
Considering that memory bandwidth is the most important for inference for most neural networks, that matters more.
| root | parent
{"deleted":true,"id":50372347,"parent":50371651,"time":1793831105,"type":"comment"}
| parent | next
{"deleted":true,"id":50371256,"parent":50371207,"time":1793825161,"type":"comment"}
dist-epoch | parent
Modern inference stacks use different GPUs for decode versus prefill. This is a GPU suitable for prefill.