Hacker News new | past | comments | ask | show | jobs | submit
ericd | on: Tinybox – A powerful computer for deep learning
Was that cheaper than a Blackwell 6000?

But yeah, 4x Blackwell 6000s are ~32-36k, not sure where the other $30k is going.

bastawhiz | parent | next
I bought the A100s used for a little over $6k each.
ericd | root | parent
Oh, why'd you go that route? Considering going beyond 80 gigs with nvlink or something?
segmondy | parent
folks have too much money than sense, gpt-oss-120b full quant runs on my quad 3090 at 100tk/sec and that's with llama.cpp, with vllm it will probably run at 150tk/sec and that's without batching.
loading story #47477967
Aurornis | root | parent | next
> gpt-oss-120b full quant runs on my quad 3090

A 120B model cannot fit on 4 x 24GB GPUs at full quantization.

Either you're confusing this with the 20B model, or you have 48GB modded 3090s.

loading story #47477468
amarshall | root | parent | next
You're almost certainly (definitely, in fact) confusing the 120b and 20b models.
loading story #47477465
ericd | root | parent
How're you fitting a model made for 80 gig cards onto a GPU with 24 gigs at full quant?
zozbot234 | root | parent | next
MoE layers offload to CPU inference is the easiest way, though a bit of a drag on performance
ericd | root | parent
Yeah, I'd just be pretty surprised if they were getting 100 tokens/sec that way.

EDIT: Either they edited that to say "quad 3090s", or I just missed it the first time.

segmondy | root | parent
you are correct, I did forget to add quad. you should join us in r/localllama

check out what other people are getting. you're welcome.

https://www.reddit.com/r/LocalLLaMA/comments/1nunq7s/gptoss1... https://www.reddit.com/r/LocalLLaMA/comments/1p4evyr/most_ec...

ericd | root | parent
Thanks for the confirmation, wasn't sure if I was just going a bit senile heh. Yeah, I love /r/localllama, some of the best actual practitioners of this stuff on the internet. Also, crazy awesome frankenrigs to try and get that many huge cards working together.

I was considering picking up a couple of the 48 gig 4090/3090s on an upcoming trip to China, but I just ended up getting one of the Max-Q's. But maybe the token throughput would still be higher with the 4090 route? Impressive numbers with those 3090s!

What's the rig look like that's hosting all that?

Havoc | root | parent
He said quad 3090 not single
ericd | root | parent
Yeah, pretty sure that was edited in after I commented because 150 toks/sec was also new, but could’ve just missed it.