What kind of hardware do you need to run a useful LLM at home? To train one?
> "What kind of hardware... "
At home, I mostly use openrouter.
A few bucks will realistically last you forever. I also bought a DGX Spark at launch. I've learned a ton. but it's not usually my daily driver. I do run overnight tasks on it, like "read this book and write up notes on each chapter..."
In retrospect, given the dirt cheap prices of apis and or renting hardware, buying hardware for inference was likely a mistake (but not enough of a mistake to sell it now).
There are a lot of tasks I have, e.g. integration with homeassistant, that I absolutely will not trust a cloud provider for. On prem or nothing.
I'm not willing to spend thousands for those features, so my house will remain dumber than it could be until prices come down in a year or two.
Schaefer says they write their own harness, not necessarily training or running a local model.
Take a look at the docs for https://github.com/niko1221/strata for some ideas, particularly https://github.com/Niko1221/Strata/blob/main/docs/MODELS.md#... .