> The idea is that we get specialized models that are better then general purpose models. But its rare for a specialized model to beat a strong general model.

Are you sure about that? I mean, MoE is basically an array of specialized models. This beats single-model general purpose performance.

Clever routing systems also seem to indicate frontier or frontier-beating capabilities at reduced cost, such as Fugu.

Further to this though, I think it hasn't been pushed hard enough. We're not going down far enough in the abstraction chain. Most peoples idea of 'specialized models' is an LLM trained on a particular domain (like math vs coding vs creative writing), where-as it should be going lower-level than that and incorporating not only more abstract and smaller specializations of LLMs, but entirely different types of AI models, classifiers, etc.

I also think the reason we don't really hear much about this (yet) is because there's no incentive for the best AI researchers/labs to go down this direction because it's better suited for local-processing over your entire codebase, but the labs all want you to perform inference on their servers and spend your money there, rather than spend your own compute to increase intelligence. They have a larger incentive to try and scale in different ways. We'll have to rely on open-source for this one.