Building the middle system that can take any model as a back end, based on cost/performance/client choices/etc, and API style present the same interface and functionality to end users is huge right now.

Local LLM usage is bouncing across major runtimes, custom runtimes and forks plus custom quants and variations of open models. llama-swap and some other tools integrate those and provide metrics but as the field expands can't keep up even with AI support.

I need to look at tailscales dashboard/etc again. I started on that when it launched but couldn't get my runtimes to go through it. I think the easiest tie in for metrics/etc is grafana + Prometheus but you need that stood up, and then a custom parsing job for each runtime based on metrics it reports and processing that data into useful dash.

I would love to see more standards here like OpenAI endpoints with metrics and information to link into. ./metrics is great but not standardized.