> LLM's power comes with it being a monolith.
Does it? All of the latest frontier models that have made technical details publicly available are using MoE; smaller specialized units that work together.
Yeah but they're all trained together along with their gate tensor. Evaluating a moderate length prompt pulls in most/all of them.