Hacker News new | past | comments | ask | show | jobs | submit
asdff | on: Apple has new Mac app coming to make running local AI models easy
Why is there a desire to have this stuff on the edge on the engineers lap instead of just building some arbitrarily scalable cluster for I'm guessing cheaper than apple's usual premium? Habitual practice from the engineers who would compile on the device getting hot in their lap? Lack of widespread hpc experience compared to running docker?
aurmc | parent | next
Many users would prefer to use a Sonnet-level (or better) LLM without sharing their data with others.

And with a device like this, you pay one single time to buy the product and then never have to think about paying for tokens ever again. For ordinary users that's silly, but for users or businesses chewing through lots of tokens, this means they can create their own turnkey replacement for "the cloud" where the only cost after purchasing the machine is the electricity to run it (which with these machines is very little). Mix that with the privacy benefits and suddenly it makes a lot of sense.

asdff | root | parent
Seems everyone in this thread is assuming I am saying go to private third party compute. No. Build your own cluster on metal you own on property you own, and have your team use that and just some thin clients to write with. Your compute goes way further. Your cluster can be going 100% of the time with jobs queued up by your team. All that memory you buy for your team in macbooks is idle and unavailable when your employee closes the lid and tosses it into the bag into the car on the way home. It could be used 24/7. On hardware you own. For cheaper to build I'm sure than apple hardware tax. On easier OS to manage and no doubt deal with anything involving any sort of heavy compute work than OS X. Are we really going to waste resources with docker when we could be using a node completely? Come on.

Like I said, seems HPC is lost art. All four sub replies missed that in my original comment.

loading story #50450544
loading story #50450423
loading story #50450403
_ph_ | parent | next
First of all, it seems to be possible to build a scalable cluster from TB5 Macs. They don't have to be on individual desks, they could be hosted in a data center.

But on the other side, equipping data centers was never Apples main business. Their approach to markets is to design products for the individuals. They also usually have very long time plans. Look at where the Apple Silicon today is and how long a path it took. What about the day, where at least a medium sized Mac becomes a great platform for running large models locally? Very easy to use, no setup, no privacy concerns. Also never worry about "tokens". You can then use AI as you use other local resources of the hardware you own today.

Gigachad | parent | next
I don’t want it in my laptop, I want it in my mini server rack at home. I’m not running any of my personal data through an external service that is certainly collecting it as training data.
usrnm | root | parent
I want it in neither my laptop nor my home server, which I don't even have, I want it in my phone or something even smaller. I understand that it's currently not possible, but this is the direction I want the technology to be moving into.
loading story #50450828
bix6 | parent | next
People don’t want to send all their private company data to the model providers?
bronco21016 | parent | next
Because why not? If you can get unlimited inference with roughly Opus 5.5 intelligence for $10k that easily beats current API pricing. Like not even a question.
codebje | parent
Distrust in allowing an unreliable vendor to control the fate of your business?