Hacker News new | past | comments | ask | show | jobs | submit
segmondy | on: IBM CEO says open weight models are now good and cheap on-prem
My local model never says no. I never get banned. If I want to do security stuff, reverse engine, there it goes! Do I wanna chat about politics? No problem! Do I want it to analyze medical stuff? I don't get to hear that it's not a doctor and I should call 911. Do I want to explore legal docs, no problem! There are lots of things that cloud models will say no to! I read that don't even agree to help with advanced biology and chemistry topic much. I also never run out of tokens, no 5 hr limit. I can run it 24/7. I don't run out of tokens. I can run 16 sub agents at the same time, non stop! 1-4 billion tokens a month? No problem. Any agent harness I want! No restrictions. Privacy all day. My own true system prompt, not something someone has determined, no filters. Furthermore, there are many open weight cloud models that don't have equivalent online. For a while before cloud models got good at math, the best model at math was DeepSeekv2 Math and there was no API. The only way to use it was to run local. All sorts of audio models, image/video models, etc. No multiple subscription needed. Private UI for my family, kids included (I can guard what kids infer). Got a friend that really needs some tokens? I give em access through private VPN. Telling someone that they are paying for decades of Claude for local AI is like telling someone they are paying for decades of public transport when they buy a car. Sure, you are right, but ...
Gareth321 | parent | next
Also no political censorship, and it doesn’t report you to the police when it suspects you did something illegal. That really happened, by the way.

Now I want a local LLM server :( $25k is just so much money.

ThunderSizzle | root | parent | next
If you already have a machine, throwing an r9700 in it is one of the cheapest ways to get qwen3.8 27B Q6 with 100k+ context working.

Sadly, the r9700 has gone up in price, but it's still the cheapest route. I use It on an old x99 Linux desktop and it achieves 50 tps decode and 1000+ tps prefill.

Gareth321 | root | parent
Thanks for the tip! I've already got a home server set up. Maybe this is my next AI purchase. Still, US$2,600 here in Europe is not cheap :-/
Jnr | root | parent | next
No political censorhip? Try asking Chinese models about Taiwan.

Some censorhip and bias towards certain ideology is baked into those models.

While it is not a problem for programming tasks, I would never use it for certain other tasks.

Gareth321 | root | parent
> No political censorhip? Try asking Chinese models about Taiwan.

Self-hosting means being able to use abliterated models where the superficial instructions like not describing Taiwan as independent are burned off.

Censorship and bias are related but not synonymous. I agree that it's harder to remove political bias when the training data includes it. We know many of the frontier models are heavily trained on Reddit, for example, [which likely explains why we see such heavy left wing bias.](https://trakkr.ai/bias/findings)

More citations:

https://www.gsb.stanford.edu/faculty-research/working-papers... (article: https://news.stanford.edu/stories/2025/05/ai-models-llms-cha...)

https://www.tandfonline.com/doi/full/10.1080/29974100.2026.2...

https://ieeexplore.ieee.org/document/10817610

https://www.washingtonpost.com/technology/interactive/2026/0...

https://link.springer.com/article/10.1007/s11127-023-01097-2

loading story #50242604
loading story #50242061
loading story #50242181
_ache_ | root | parent | next
I'm building a local LLM server myself. For about 8k€ you can have 170tps, ~6000 prefill.

A lot more with Haiku equivalents (Qwen3.8 27B) or gemini. It's not frontier, but not that far away.

Gareth321 | root | parent
I would love to hear more about your setup. Which hardware and models? How much context can it handle? How does it compare to frontier models?
loading story #50242378
InsidiousWombat | root | parent | next
renting GPU off cloud is also an option
cft | root | parent | next
MacBooks with large RAM and optimized for local models should be Apple's medium term goal.
Gareth321 | root | parent
It's a good first step. They're not great but they kind of work for many tasks. I think the small models can be useful in context-specific tasks but the friction is high right now because the user is forced to determine whether Siri is good enough, or a better LLM is required per task. Ideally Siri would be smart enough to intelligently route. Apple claims that's how it works but even their cloud processing is far behind frontier models.

Worryingly, they're taking a walled garden approach, and they don't want to expose user data to competitors. After the monumental failure of Siri over more than a decade, I don't trust Apple to execute competently on AI.

charcircuit | root | parent
>and it doesn’t report you to the police when it suspects you did something illegal

This is an alignment issue. OpenClaw with a local LLM with an email you setup for it could contact the police.

Gareth321 | root | parent
It will not do that unless you specifically teach it to do that, or you use an Anthropic model. The issue in this case was Anthropic's safety monitoring built into the model. They record everything and flag potential illegal activity for human review. This occurs even over the API. I do not know if OpenAI does this, but I would not be surprised. I think the Chinese models are far less likely to do this (unless one resides in China).
sharmajai | parent | next
A cloud model can also get neutered anytime and/or start using more tokens while charging you the same or more per token.
ceejayoz | parent | next
> My local model never says no.

> I don't get to hear that it's not a doctor and I should call 911.

https://en.wikipedia.org/wiki/Darwin_Awards

mik1998 | root | parent
It's pretty easy for a reasonable person to determine when to call 911 and when to ask an LLM. Typically when I ask an LLM, it is for something I am not willing to escalate further at the moment.

Anthropic has a significant interest in reducing liability, which in this case means mostly blocking this use case.

loading story #50243711
jacquesm | parent | next
Exactly, that's the spirit. I pool the resources with some friends as well.
latentsea | parent | next
> My local model never says no. I never get banned.

So much this. Plus everything else you mentioned. The freedom and control is real.

| parent | next
{"deleted":true,"id":50240784,"parent":50239362,"time":1793003331,"type":"comment"}
jack_pp | parent | next
isn't it the same using the same model you host locally but through a cloud provider? does the $ math favour a local setup then?
Forgeties79 | parent | next
I think both of y’all glossed over the $25k of it all pretty quickly. This isn’t remotely an option for anyone.
bityard | root | parent | next
You see it as money spent, others might see it as an investment. OP doesn't necessarily say exactly what they use it for, but if education is the goal, $25k and spending your evenings learning about AI theory and tooling is way cheaper than a university degree and can easily result in millions more dollars in lifetime earnings given the current early state of the technology.

Besides, I regularly see people here in HN spend a LOT more than that on hobbies and depreciating assets. A few years back, people with coastal salaries and tech jobs were tripping over themselves to spend $100k on a certain brand of trendy electric car.

solyin | root | parent | next
AnyONE. But if you got some friends, and they got friends, then that $25k becomes smaller and stomachable. My favorite rumor out of LA is groups of friends with a Lamborghini Urus that they timeshare. No idea if it's true, but if car people can get together to buy/lease a car that's MSRP is $345k, I think $25k should be an option for anyone with friends. (Hypothetically. I've read a lot about this friend thing on the Internet!)
sailfast | root | parent | next
But I don’t need my LLMs part time? And we all live in the same time zone.

And what happens when the girl at the bar figures out that you TIMESHARE that Lambo?

loading story #50242219
Forgeties79 | root | parent
If 10 people are sharing it the speed advantages suddenly drop drastically. That’s $2500 a head. 5 would be $5k. Then all the rules and such you need in place

This is all this stuff that dries people to pay for frontier models. Even running modest models requires a few thousand, unless you just want really lean ones. People don’t need frontier but it’s not cheap to effectively run, say, a 16gb Qwen 3.6 or 3.8 in the 30b range. Ram is like $15/gb and GPU’s with 16gb vram are hardly cheap.

loading story #50243615
segfaltnh | root | parent
Its a drop in the bucket for many businesses though.
| root | parent
{"deleted":true,"id":50241238,"parent":50241186,"time":1793007051,"type":"comment"}
charcircuit | parent
>I don't get to hear that it's not a doctor and I should call 911.

Local models can do this and with tool use they can even call emergency services for you on your behalf.