Hacker News new | past | comments | ask | show | jobs | submit
Gareth321 | on: IBM CEO says open weight models are now good and cheap on-prem
Also no political censorship, and it doesn’t report you to the police when it suspects you did something illegal. That really happened, by the way.

Now I want a local LLM server :( $25k is just so much money.

ThunderSizzle | parent | next
If you already have a machine, throwing an r9700 in it is one of the cheapest ways to get qwen3.8 27B Q6 with 100k+ context working.

Sadly, the r9700 has gone up in price, but it's still the cheapest route. I use It on an old x99 Linux desktop and it achieves 50 tps decode and 1000+ tps prefill.

Gareth321 | root | parent
Thanks for the tip! I've already got a home server set up. Maybe this is my next AI purchase. Still, US$2,600 here in Europe is not cheap :-/
Jnr | parent | next
No political censorhip? Try asking Chinese models about Taiwan.

Some censorhip and bias towards certain ideology is baked into those models.

While it is not a problem for programming tasks, I would never use it for certain other tasks.

Gareth321 | root | parent
> No political censorhip? Try asking Chinese models about Taiwan.

Self-hosting means being able to use abliterated models where the superficial instructions like not describing Taiwan as independent are burned off.

Censorship and bias are related but not synonymous. I agree that it's harder to remove political bias when the training data includes it. We know many of the frontier models are heavily trained on Reddit, for example, [which likely explains why we see such heavy left wing bias.](https://trakkr.ai/bias/findings)

More citations:

https://www.gsb.stanford.edu/faculty-research/working-papers... (article: https://news.stanford.edu/stories/2025/05/ai-models-llms-cha...)

https://www.tandfonline.com/doi/full/10.1080/29974100.2026.2...

https://ieeexplore.ieee.org/document/10817610

https://www.washingtonpost.com/technology/interactive/2026/0...

https://link.springer.com/article/10.1007/s11127-023-01097-2

schiffern | root | parent | next

  >abliterated models where the superficial instructions like not describing Taiwan as independent are burned off
Does this help with issues like including security vulnerabilities when you say you're in Tibet? I understood that this was burned in at the training level, not just a layer of censorship added at the last minute.

https://www.crowdstrike.com/en-us/blog/crowdstrike-researche...

loading story #50242731
Jnr | root | parent | next
All good points, but I am not sure who I trust less. Chinese companies or random people on Hugging Face. :) It is still a black box. And I have a feeling I could not abliterate those models on my own hardware.

I am not saying I would not use any of them at all, but I would avoid doing certain tasks with them. And with the testing I have done with abliterated models in the past, it is not 100% successful and it can still hit the same walls as original model.

loading story #50242763
xeyownt | root | parent
So... the Super Intelligence is Democrat?
loading story #50242531
_ache_ | parent | next
I'm building a local LLM server myself. For about 8k€ you can have 170tps, ~6000 prefill.

A lot more with Haiku equivalents (Qwen3.8 27B) or gemini. It's not frontier, but not that far away.

Gareth321 | root | parent
I would love to hear more about your setup. Which hardware and models? How much context can it handle? How does it compare to frontier models?
_ache_ | root | parent
I mostly run Qwen3.8 Flash Next on 2xR9700 (but Intel Arc Pro B70 may be better currently, or 4x3090 is even better but out of budget, 32Gio Nvidia GCs on ebay are also usable). I have 128Gio DDR4 (3200Hz), but 64Gio is enough, even 32Gio maybe, I just reuse it from an old project.

You may need a AMD EPYC processor and associated motherboard, don't forget a good PSU at least 1200W (300W processor + 400W by card to support spikes of consumption). On some setup (~15k€) you can use 4 CG to go up to 128G of VRAM.

I'm not using a EPYC processor for now, so I get 170tps on decode but very slow prefill, the bottleneck is the PCIe exchange between the cards, I need to setup P2P (it means what you think it means, for GCs).

My setup supports full context (2*18 bytes so 256K), I didn't setup YaRN to get 1M of context (2*20 bytes), but it's an option with 128Gio of RAM, the visual component is extracted from RAM so a little slower. I'm not using multiple agents but some people does, it's god like with 4 graphics cards, on 2 only, it's relatively quick but limited to single user and up to 3 to 4 concurrents requests only.

It's not frontier, but look AA of QFN, it's good enough. DeepSeek V4.1 can run on my setup, but it's slow. On dual B70, there is some PoCs to make it usable (>60tps). It improves every days.

If you need more information, look for local setup users on the Web. There is a lot of people sharing information about benchmarks and setups. L80 is my reference for now.

loading story #50243260
InsidiousWombat | parent | next
renting GPU off cloud is also an option
cft | parent | next
MacBooks with large RAM and optimized for local models should be Apple's medium term goal.
Gareth321 | root | parent
It's a good first step. They're not great but they kind of work for many tasks. I think the small models can be useful in context-specific tasks but the friction is high right now because the user is forced to determine whether Siri is good enough, or a better LLM is required per task. Ideally Siri would be smart enough to intelligently route. Apple claims that's how it works but even their cloud processing is far behind frontier models.

Worryingly, they're taking a walled garden approach, and they don't want to expose user data to competitors. After the monumental failure of Siri over more than a decade, I don't trust Apple to execute competently on AI.

charcircuit | parent
>and it doesn’t report you to the police when it suspects you did something illegal

This is an alignment issue. OpenClaw with a local LLM with an email you setup for it could contact the police.

Gareth321 | root | parent
It will not do that unless you specifically teach it to do that, or you use an Anthropic model. The issue in this case was Anthropic's safety monitoring built into the model. They record everything and flag potential illegal activity for human review. This occurs even over the API. I do not know if OpenAI does this, but I would not be surprised. I think the Chinese models are far less likely to do this (unless one resides in China).