Non-forking httpd servers using select() were a popular little coding challenge for a while in the 90s. Spinner was one of them.
Nginx’s real strength was being able to proxy and cache HTTP using that same mechanism, so you didn’t additionally need to deploy Varnish or some other appliance.
As to whether this is a good mental model for what is coming for local LLMs, I am not sure I am convinced. Apart from more quantisation-aware training, perhaps binary and ternary aware training, custom inference engines per model, and maybe some improvements in diffusion models, the grand challenge in small footprint LLMs is training really small reasoning and tool use models, and so far it’s far from clear they can deliver.
Truly tiny models will not be viable as general coding assistants; even 12B dense is too small and you will find plenty of people who will tell you that 26B/4B or 35B/3B MoE is too. Though perhaps they can be trained for single languages, like just Python or just TS/JS.
More likely is the idea that 30-40B dense models might be good enough for most things once low cost and likely bespoke hardware catches up.
But I don’t think any truly profound advances seem likely in software or training alone. I am no expert but it feels like we’re already a lot closer to efficiency than we were in your analogy, and the gains are perhaps not going to be much more than small increments.
Maybe we will see something like a ternary 60B/10B MoE model turn up. But at the moment at least I am not sure where the incentives are to train these.
Fun fact: Igor Sysoev originally wrote mod_accel [1], an acceleration and reverse-proxy caching module for Apache before he made nginx.
I remember using that module in the mid-2000s as a load balancer (and to offload SSL encryption/decryption as it was a CPU-expensive operation).
My naïve impression is that the LLM world will keep delivering these fractional improvements for some years at the cost of simplicity. And sure, ontological support seems quite promising.
But making things radically better or faster for small models in the way that is hypothesised, that feels like it can only come as a result of hardware performance improvements and likely architecture changes.
Because there's no free lunch, right? Speculative drafting for example, noticeably improves performance until acceptance rates drop for reasons that have to do with the particular application, and then it starts hurting you, especially near the limits of the memory bandwidth. Because once it is wasteful it is an extra overhead.
I gather Gemma 4 supports, in principle, dynamic speculative draft lengths, to help with this — where it will stop making bold predictions when the success rate falls. But I'm not sure if any of the inference engines I've tested with support that.
I think small models are miraculous — I still think Gemma 4 12B is astonishing — but I guess what I am saying is that I think maybe technology is moving quickly enough that the developers are done with the low-hanging fruit.
Even if the idea of "no free lunch" was taken liberally discounting the cost of research, it would only be limiting to systems built from a foundation of optimization, but that's not the case. The foundation so far has been one of brute force scaling. Usually meaning there is lots of room for optimization.
In cloud AI, sure. But in the smaller open weights model territory it feels like we're well into optimisation?