I have 3 3090s, all in various gaming computers. I’m not sure what to do with them- the VRAM is only 24GB so doesn’t seem worth the power costs to run an open weight model. Any suggestions on how I can run them as some type of cluster? Is it worth exploring NVLink?
Club 3090 will make installing llm easier, for 1 or more cards.
You can layer models across multiple GPUs for inference. If you pooled it into one system, that would be 72gb of some pretty damn fast memory. Would be great for dense models in the 30-50b range.
You can put dual 3090's into one system.
https://www.reddit.com/r/LocalLLaMA/comments/1m5fkts/my_prac...
[dead]