Particularly, I had one team member who was extremely sceptical of AIs/LLMs/harnesses and refused to use them. One day he said "Well, I have an RTX 5090 doing nothing... should I try to get something up on it?" and a few minutes later he had 3.6-35B loaded up, running OpenCode.
It continues to be a workhorse to this day, running on both my local Mac for various types of jobs, an AMD R9700 at the office, and said teammember still uses it on his 5090, although in practical terms we do a lot more with DS-V4-Flash-0731 these days.
I haven’t found it very useful for code. It can do some code, but I’ve tried a dozen different quants and context lengths and the output is always bad enough that it has to be discarded for anything other than really easy tasks. It has been useful for exploring codebases for search and summary, though.
DS Flash is where local models begin to feel useful for coding, but the quants we run locally are sharply reduced in intelligence from the benchmarks for the full models.
For applications where data cannot leave the local network it’s good to have them. For actual coding work I can’t actually justify the power of electricity and cooling, let alone the expensive hardware, compared to hosted APIs.
But I admit I do enjoy playing with them anyway. I think it’s one of those hobbies where it’s most fun if you never do the math on how much you’re paying for the privilege. If someone has a requirement that data stay local then it’s different, of course.
Do you mind sharing your use cases?
I started with Karpathy's LLM wiki, and did everything he said not to do - downgraded the model to mere tool usage and summarization, and it works great.
I am a data hoarder, and finally I can just dump all the content I remotely like, and get something interesting to browse for the price of electricity.
Agentic long-running tasks, as others have mentioned:
- Groom and triage tickets for agentic SWE workflows
- bug hunt — the probability of Qwen fixing a complex bug is 50/50 but often it is capable of identifying the root cause or at least laying the ground work for a more capable model to pick it up.
Sure a lot of this could be done without AI, but it's certainly quicker and easier, and since my AI box is on solar, it's just the power of the sun to keep it going.
The A3B models are super fast but I found the A3B Q4 model ran in circles a lot and ended up taking longer to complete tasks that 27B Q6 because it kept having to redo/rethink/fix something.
I was writing extensive prompts to rein it in and it would still ignore basic directives like "never force push on the repo, ask me instead". I ended up switching back to 27B after about a week of frustration and lost productivity.
Gemma QAT is an honourable mention.
- You at least need a capable machine, so that's not 0 monetary investment. - You need to spend at least an hour decicding between ollama, llamacp, mlx, etc. - You need to find the correct quantized version of the model that works for you based on the architecture. - You need to figure out the correct context window size to get reasonable performance. - You need to setup a harness that works against your model - You might need to setup additional websearch tools, image tools, etc since harnesses like pi don't come with the model. Ofc you can't use codex and claude code, because those aren't opensource and you are anti-AI.
Or, you could sign up for Opencode for $10 and just be productive.
I'm particularly calling out the hypocrisy of the original comment. Being Anti-AI, and then spending hours on setting up a less than frontier AI model.
You forgot the step before where you spend months waiting for security to vet it, legal to sign off and procurement to approve it.
Or you could use hardware your team has lying around. Everyone isn’t working on cloud-hosted CRUD APIs.
Local AI is almost perfect. But its like all democracy: its history is marred with lots of crap.
The original comment states that the person in question already had a suitable graphics card to hand, so it did not require a monetary investment.
GP clearly states that "someone who is into that that stuff" was guiding the process, so it did not require a significant time investment.
> I'm particularly calling out the hypocrisy of the original comment. Being Anti-AI, and then spending hours on setting up a less than frontier AI model.
I see no hypocrisy in the original comment.
You've also assumed the skeptic in question doubts the capabilities of AI. That may be the case (like you, I have no idea), but they may also have privacy concerns, in which case a local model is the appropriate choice.
There are plenty of reasons to be skeptical of AI.
For me at least the local AI stuff, powered with solar has been pretty great. Would that scale to a large business? Goodness no, but for my tinkering and learning, it works great.
It is 0 monetary investment if I already have said machine lying around doing nothing.
Which is exactly the story OP talked about.
Pi has a nice guide on it (https://pi.dev/docs/latest/llama-cpp) and it is really not that hard.
How is that hypocrisy? Self hosting is somehow anti AI? Its not anti AI. Its literally using AI!
…and honestly, at a higher technical level than slapping your wallet against a token provider and running prompts in a hosted sandbox you can't even see the prompts in.
Local: You need to invest $thousands into GPU and/or very-high-end CPU+Memory hardware.
Vendor: You can use any existing device, even a phone or tablet. A very low-end laptop is fine.
> takes literal minutes to get started
Local: Typical scenario is hours just to download the software, the model weights, and then faffing around with CUDA and matching your GPU drivers.
Vendor: Free-tier available instantly on a web URL. Even local agents have free tiers from multiple vendors. Install is a single command and/or download and "next,next,next,finish" wizard that takes ~1 minute.
> you can just `rm -fr` it and forget the whole thing existed.
I'm still cleaning up multi-GB model weights floating around in hidden subdirectories under my user profile from months ago when I was experimenting with local models!
Meanwhile I simply... stopped using Gemini. That was the entire process: I no longer actively use it. They stopped billing me for my token usage, because it is now zero. That's... it.
You have it totally backwards.
Are you trying to say that local models are hard to use because... you're having issues handling files properly? I am not sure I get the argument.
I get the rest of the comment: local models require an investment upfront, and it is less convenient. It doesn't say that it is not cheaper, though.
I guess they were using ollama, which does not tell you where it puts the models it downloads.
I used to deal with these kinds of frustrations too.
fd --unrestricted --size +1G
fd --help
-u, --unrestricted...
Perform an unrestricted search, including ignored and hidden files. This is an alias for
'--no-ignore --hidden'.
-S, --size size
Limit results based on the size of files using the format
<+-><NUM><UNIT>Same story on an RTX 4060 Ti 16 GB. MTP is a serious boost to tg.
Downloading the model is a simple hf command that HuggingFace's web UI even gives you.
llama.cpp is trivial to use, and so is llama-swap, if you want to use other models too.
If you don't know what arguments to run it with, you download ggrun and use that.
Local LLMs are incredibly capable and don't need expensive hardware. A $500 GPU will do. Or even cheaper.
This is all trivial.
[0]: https://www.reddit.com/r/LocalLLaMA/comments/1vef79c/quantiz...
Download LM studio, search models, click download, wait minutes, prompt and have fun
That's one of my main issues with AI anyways, the thought of having all my data go through some sketchy foreign (to me) entity with questionable motives and under a questionable regime.
Local AI solves for all of those.
If you're learning about model inference, then it's a different and you are definitely not anti-AI in that case.
It's not that billing is complicated, but learning to set up a local AI is a lot more useful and more rewarding.
Isn't that just the definition of MoE vs dense ?
I honestly think that with my electricity prices running qwen 36B myself is more expensive than hitting the cache rate at deepseek.
I gave it a try for a few days (pi + openrouter + deepseek-v4-flash via deepinfra) and ended up paying ~$18 for rather light usage. Yes it's still cheap, yes it's fast, but i feel i would still get a better deal with a Claude subscription plan.
Kind of slow, but using only 14W of electricity so the Wh per task is twice as good as using my i9-13900 laptop with 4060 GPU.
Tolerable and usable for some things, though thinking makes it take about a minute to reply in many cases. But getting this kind of thing to run on 8GB of VRAM is the main benefit of the mix-of-experts setup: it can do partial GPU loading and get a ton better throughput than a similarly-sized dense model (like 5-10x, sometimes more).
The weaker point is prompt prefill, which starts at 1,400 tokens/sec but decreases significantly at high contexts. That said, for agentic scenarios, if you're using a harness that doesn't needlessly bust the cache, it doesn't feel slow.
I really hope they release a Qwen 3.8 35B, although the lack of a mention seems ominous.