Hacker News new | past | comments | ask | show | jobs | submit
I really do not understand the value of running large models locally. I am absolutely fond of local models but for specific tasks. The agents need to do a lot of work in this case: orchestration, big or small model split of planning and work, split into small tasks, merging, etc.

LLMs with <30B parameter models seem like the sweet spot. I would rather focus on multiple tiny LLMs that are tuned to specific tasks (SQL queries, finance/accounting, backend/frontend dev, etc.).

For larger models, a $10-40 ish Neuralwatt/Opencode Go or any similar subscription goes so far for the average user that it is ridiculous. We are still in the "youth" era of agents. There are many patterns that work well with smaller LLMs. Or with a big-small LLM combination.

My context:

I own the tiny 16GB M4 Mac Mini and I use LLMs (coding agents) to build software and a few of my products use LLMs directly. Full-time user of LLMs, do not read or write code manually. I run the tiny Qwen series (<10B params) sometimes but a couple $20 subscriptions (Claude, Neuralwatt) take care of my full-time agent usage.

loading story #50450942
loading story #50450649
Model/harness that runs locally, which you can connect from your phone to do non-dev queries (looking stuff up, recommendations, planning and etc.) can save ~$300/year on various subscriptions.
loading story #50450501
I have the same Mac M4 16gb. Do you have the Qwen model assist the larger models for menial tasks/tool calls during coding it can handle? As in have them work in tandem to save on tokens/calls on the large models or are your local agents just meant for their own specific side tasks?
loading story #50450536
loading story #50450733
The addressable market is enthusiasts. The cheapest and least loyal consumer that exists. But I guess it is just a small side project for apple which is fine.
loading story #50450658
loading story #50451222
loading story #50450627