Hacker News new | past | comments | ask | show | jobs | submit
somenameforme | on: GLM-5.3: Frontier coding with emergent cyber capabilities
Another interesting potential market here will be 'LLM in a box'. All the hardware and other tooling in a prebuilt, but modular, package ready to go. Pay one up-front cost, get a system running [whatever open LLM] with a token rate of [x], optionally configured to be immediately ready for distributed usage. Basically the opposite of cloud stuff: no rent, no dependency, 100% guaranteed uptime, guaranteed security/privacy (at least subject to your own actions), and so on.
adrian_b | parent | next
Palantir already offers a "turnkey AI datacenter", i.e. a rack with "NVIDIA Blackwell Ultra systems with eight NVIDIA Blackwell Ultra GPUs and NVIDIA Spectrum-X™ Ethernet networking for AI training and inference".

It is said that it comes with all hardware and software required to run inference or training with an open weights LLM.

The existence of this product, which competes with cloud-based offerings like those of OpenAI and Anthropic, is presumably the reason why the Palantir CEO criticized very harshly some time ago the business model of OpenAI/Anthropic.

While I doubt that the ethics of Palantir is any better than of OpenAI/Anthropic, in this particular case I have to agree with Alex Karp about "Sovereign AI", i.e. that only losers will make their business completely dependent on an external entity like OpenAI or Anthropic, who are certainly not trustworthy.

vrganj | root | parent | next
I'm not sure a data center run by ... Palantir of all organizations is what people have in mind when they worry about data sovereignty.
adrian_b | root | parent | next
They are selling it, not running it.

It is just a dedicated computer system, which should be managed by its owner, like any other on-prem servers.

I doubt that it has a good price/performance ratio, but it is a solution for those who feel that they do not want to search, buy, assemble, install and configure every HW/SW component.

iinnPP | root | parent | next
I take it we saw different demos.

I'm under no NDA, if you actually want to know what's up.

oarsinsync | root | parent | next
I'm assuming you're alluding to them selling a managed solution, alongside the unmanaged solution that the GP is referring to?
XVII | root | parent | next
I want to know, please tell us
| root | parent
{"deleted":true,"id":49301722,"parent":49296929,"time":1786727848,"type":"comment"}
idiotsecant | root | parent
Ohhh spooky vaguepost.
Panoramix | root | parent
For those that don't mind a lot of rootkit and embedded spyware you mean
crossroadsguy | root | parent
Fair but the idea of "running your LLM setup" at every "need" level and corresponding cost does make sense.

For a lot of people (and orgs I'd guess) who just go and buy ≈$20 per month plans (or more for teams), they might not even need a fraction of that cost or capability. A lot of them don't even need it for coding or graphics. Even the API access based pricing aren't great from these frontier US AI houses. The distribution of "LLM being" offered will also give rise to many open-router like offering but at the end point level - direct interfaces to the customers. Pick your vendor sort.

AI shouldn't become another "search means Google".

rxyz | root | parent
How is this different from buying a supermicro rack? Better support?
bevekspldnw | parent | next
“100% guaranteed downtime when you least can afford it and the support tickets are your problem.”

We’ve a hybrid shop, including hosting our own ML infra, and we save a ton from cloud spend with local ML. Easily one million USD over past three years. But it’s not “free”, you are shifting a lot of labor into your plate.

hypfer | root | parent
And with that also gain institutional knowledge, skill up your workers and attract talent that wants to work on this stuff.

All boils down to short-term/long-term thinking.

gruturo | root | parent
This. People WANT to work on this stuff. And having skilled workers is a precious advantage.
bevekspldnw | root | parent
Still has to break even on the balance sheet, especially at a bootstrapped startup. We actually made most of the financial windfall in translation API fees oddly enough.

For our own model training we needed to do some large scale translation tasks of a large dataset (1M or so documents, 10 or so target languages), running full-size NLLB on-prem saved us an absurd amount of money vs Google Translate API.

(For reference doing 1M target docs into a single language in Google Translate API is roughly $120k list price. You can run full size NLLB on an 48GB NVIDIA A600 and the major difference for us was speed, but for this task time to completion wasn’t an issue.)

jurgenburgen | parent | next
> 100% guaranteed uptime

Disagree there but I think this is an interesting idea. We would need to find some more cost-efficient hardware to run it on than Nvidia GPUs.

pulse7 | root | parent
It will come... all big hardware players (Intel, AMD, Broadcom) and dozens of startups (Tenstorrent, etc.) are working on it...
numpad0 | parent | next
What makes that kinda complicated is that multi-user throughput of LLMs scale well but single-user performance often stays constant at low ends. If you could saturate e.g. 16 concurrent session-month of demand, you can just go buy 16 of 32GB GPUs and start charging monthly for inference. That could work if you had e.g. over thousand total employees with hundreds of devs eager to trying it out, but only if the company is also interested in a private inference experiment.
zozbot234 | root | parent
You're talking about multi-session vs. single-session throughput. A single user can easily leverage multiple sessions via e.g. subagent swarms, especially on a lower-end setup where any single session is going to be quite slow. Saturating utilization during off-hours is harder but potentially quite feasible by assigning lower priority, unattended tasks/inference loops.
jnwatson | parent | next
I see ads for this all the time.