Hacker News new | past | comments | ask | show | jobs | submit
segmondy | on: Tinybox – A powerful computer for deep learning
Some of you folks on here love to argue, gpt-oss-120b was trained in 4 bits, so it pretty much takes up 60gb.
Aurornis | parent
Good point, but you still need KV cache and more. Fitting the model alone to RAM doesn’t get the job done.
segmondy | root | parent
Yeah, it doesn't take much. I'm looking at it right now, KV cache is about 4gb of vram, compute buffer =~ 1.5gb at full 128k context.