Hacker News new | past | comments | ask | show | jobs | submit
zozbot234 | on: Tinybox – A powerful computer for deep learning
MoE layers offload to CPU inference is the easiest way, though a bit of a drag on performance
ericd | parent
Yeah, I'd just be pretty surprised if they were getting 100 tokens/sec that way.

EDIT: Either they edited that to say "quad 3090s", or I just missed it the first time.

loading story #47477624