When that article was first published , I started looking into llama.cpp. With the help of an LLM I researched the knobs to turn that made most sense.
Things went from "local models are useless unless you have a 512GB GPU I guess" to "oh.. I can do a lot of stuff locally now!"
Had a similar experience. Llama.cpp compiled natively; parameter sweep to find best options fitting my use case for the qwen models with 16GB VRAM. The whole thing packaged into a portable container.