Pete warden on local AI, open models, voice interfaces, the future of hardware
https://thedataexchange.media/pete-warden-moonshine/I exclusively use this setup for all AI usage from fixing obscure reproducibility bugs in software that once took me weeks, to compiler bugs, to kernel hardening and hardware drivers, to building a complete unprivileged and security hardened replacement for systemd in rust. Most of that was done using Qwen 27b back when I had only 2x r9700s.
There is literally not a single problem I have faced in 25 years of systems engineering that I cannot solve now 10x+ faster using my local LLM setup.
I turned down the free OpenAI and Anthropic tokens in my YC batch because I have absolutely no interest in being reliant on third parties to do my job.
Which is exactly why said power and utility will soon be BANNED under a myriad of false aegis'.
An economy that almost exclusively extracts ECONOMIC RENT from the working class, won't permit any "leakage" or autonomy.
Most of my peers do use OpenAI and Anthropic for development work, which pains me, but I will never tell someone else how to do their job if the end result meets my quality bar.
I lead core systems engineering building custom enclaves and operating systems and I do 100% of my work with hardware I own on my own property. When the internet goes down, I do not care and keep building.
2xDGX Spark cluster is my current load, it is sufficient for a great deal many wonderful, wonderful things. It isn't fast as consumer AI, but I have nevertheless been able to do wonderful things with it for the past year'ish.. I'd get a couple more, but indeed they've tripled in price, so yeah .. things will just do for now, and it's fine. Pretty great, actually.
I think the way we control AI is we air gap it. My cluster is air-gapped.
i just calculated that it should be Per month (30 days) 380euro in my region, which is a lot after an initial 20k.
I have to say, no limits stopping sounds nice though.
$300/mo in the bay area for 1kwh running 24/7, at least until I get solar installed, then free.
> do you not fear that your hardware will be obsolete
If it never gets better than it is right now, I cannot think of any problems in software I cannot solve locally, so I am all set for a while.
Though also, the capabilities on my -existing- cards have gone UP dramatically in the past 6 months since as the local LLM community keeps optimizing.
Sure in 3 years all these cards will be a lot cheaper than I paid, but having them for the next 3 years is absolutely worth it for me.
This costs me less than a used car. Most people chase car and phone upgrades every 2 years. I tend to get something good enough and upgrade once a decade. Depends on your goals really.
In practice though, I can produce about 3B tokens/month of Qwen 3.8 Flash Next which is the model I crank almost 24/7. That would cost something like 1k/mo on OpenRouter. My setup costs about $300/mo in electricity, though getting solar soon to drop that to zero.
My entire 12u rack has roughly the same capability as an H200, though at maybe 15k all in with mostly used parts vs 50k for a used H200 checking todays ebay prices and that is -just- the card you still need to build a system for.
Consumer cards are a lot more power hungry than enterprise cards and they take up more space. I run my setup next to my dryer, running on 240v for 15% better power efficiency.
Also using transparent memory encryption on a Xeon and an EPYC for confidential and private inference guarantees no third party can offer me. Of course I can run abliterated models that do any security research I want without refusals.
Strata and Radiance are the state of the art ATM for running big MoE models on small consumer GPUs at high performance.
Just starting to dive in on local AI with my own 16gb card, and the struggling with evaluating what tradeoffs are worth it.
As for cost a 9700 is running at about 2k USD MSRP right now and the rated power is 300w.
It seems like it just shifts which third parties you rely on. I think local inference is cool and your setup sounds dope, but you still have third party hardware manufacturers, and third party models in the mix.
What is the advantage of this over OpenRouter, which lets me select for, jurisdiction, quantization, ZDR, etc.
note: My use case can't even come close to having this make sense financially, but I understand if yours does. Either way, that doesn't make the third party argument better.
sure, in a very general sense you're dealing with third parties in both scenarios, but if you actually own the hardware and model weights, you can control your own destiny for the lifespan of that hardware.
no ongoing liabilities except the power bill and opportunity cost of dropping $20k on hardware.
He isn't using OAI or Anthropic models. He is using models that area available for absurdly cheap from completely independent providers in multiple jurisdictions.
I get the base level argument, but from a practical point of view there are so many different dependencies that shifting one isn't really doing much.
When the internet goes down, my productivity is not impacted, and I have total privacy on my work at all times. Also no rate limiting etc, and I can burn tokens freely.
If you own your own hardware, you don’t have to depend on and trust third parties continually, just once when you bought it.
I'm not the one claiming to be free of third party dependencies. Third parties built my hardware, they wrote the firmware in the hardware, they wrote the OS on the hardware, they designed the network stack that you might use to connect to a local server, they provide security updates to all of that software, they ensure that I have a stable power supply to run all this, etc.
I'm incredibly reliant on third parties, some of which happen to sell me tokens.
If the power goes out, internet goes down, I fire up my generator and keep building.
We are talking about digital sovereignty and ownership.
I own the means of my production.
They release a new RTX series every two years, last time was 2022. (edit: I mistakenly quoted the year wrong, not sure why, but it seems obvious to most that Nvidia doesn't care about the local consumer market anymore)
Meanwhile major AI companies are investing in their own chips, if Nvidia wants to survive they really should not abandon consumers.
Hoping AMD steps up to the plate. Meanwhile the nextXbox and PlayStation consoles are going to have a rough time with their next consoles which will likely cost a thousand dollars or more is my gut feeling.
Regardless, they aren't planning on sticking to the two year release cadence any more which does suck.
RTX 5080 - release price $1000, average price today $2300
Nvidia also stopped selling its own version of these cards directly to consumers and is happy with letting partners and resellers/scalpers control the price.
So it is pretty obvious that the company doesn't particularly care about this market segment.
Which I think is foolish, every AI shop is making their own alternatives, it's not going to be a winning bet for them.
Local inference gets truly interesting when the proper next-gen, natively AI and not just a repurposed GPU, stuff starts to land. Initially things like compute-in-memory and fancy flash, and later possibly the truly exotic stuff like various RMM (read-mostly-ram, for example stacked FeFET on logic) tech.
IMO there is no chance for ubiquitous local AI to happen if that means people using GPUs. But the industry is chasing all possible solutions for more efficient inference platforms, and if something like pachinko machine architecture with FeFETs ends up winning, suddenly buying a local ai inference solution in a box is no longer as insane.
(I say this as someone who would have liked a 5090...)
AMD's aggressive marketing as an AI-first company leaves me pessimistic about that.
Apple silicon says "hold my beer", which I know because i've been doing this on my 3 year old quite beefy mac studio, for example. it's good enough for many tasks.
Yes, there are a hundred valid reasons to self-host. But 99%+ of consumers aren't going to care.
People don't even care when the cost of hosting is ~free, like a simple home server. How many are ready to buy thousands of dollars worth of AI hardware, which could very well be obsolete a couple of years from now?
If we care about self-hosting and open source and decentralization because we believe it leads to a better world, and not simply because we want to be control freaks and snobs, then we need to learn how to build shit that is actually convenient and usable and understandable to normal people.
I think it's possible. I think many people would want this, and care, if we could package it up in a digestible way.
Before that number is high, can we realistically expect non-techies to use self-hosting?
Also, none of this is intuitive and a lot of popular closed stuff is harder to use (and vastly disliked by the very same normal users) than some popular open source stuff.
I'd even mention 'normies' want self hosted Word and Excel and regularly reject the web-based versions of them. Normies don't want streaming games but a gaming PC or console. Normies may buy streaming services like Netflix but get upset when their favorite shows are removed, so they actually don't like netflix. Normies complain about censorship on twitter and reddit and discord and would love an alternative they control, but may be unaware that's an option and seem stuck on those services and hate them.
Normies want self-hosted all the time too. Their ability to understand and do this is what's lacking. So that's a conversation on its own, but normies aren't actually happy. Form what I can tell most people hate 50% of the stuff they use right now and most of those complaints revolve around the cloud-hosted/lack of control/privacy and other issues self-hosting or federated services or FOSS apps could arguably solve for them.
This is a big conversation but imho tends to boil down to who has the marketing and sway to get you to install or use something, not that typical end users are savvy judges of software and useability working purely on merit and informed facts. Does your grandma actually like Facebook, or is she just lonely and its the lowest friction way to not be lonely? Do you really like Chrome or Word or is it just easy to get and popular? Is the big mac the best burger or is just easy to get and popular?
I mean...sure...but how convenient was that Office 365 outage? How convenient is it to find that a feature you actually depended on is gone because the company that runs your software decided to get rid of it?
It is always a back-and-forth. It is not as if centralized computing is a new idea, in fact it was the original way to use computers and people found that issues of control mattered and switched to PCs; the pendulum swung back because people realized that there are advantages to centralization (e.g. not having a failed hard drive destroy high-value documents). The cycle will probably continue for a long time to come because the advantages and disadvantages of both approaches are fundamental.
It’s not so much about being snobs as it is about not having time and energy for a second job. The level of quality you demand simply requires a level of attention, investment and involvement that can’t be provided without working full-time on it, for profit.
Telling a bunch of hobbyists that they should provide service of a quality equivalent to that of a trillion-dollar corporation if they are serious about it is gaslighting.
That's a mirror that hurts to look at that you just held in front of my face. Guilty on both counts, even though I try to convince myself I'm not
They don't now, but they might soon.
It used to be that bad behavior on the part of our service providers was annoying because you'd get ads and spam. Nowadays the leaked data gets aggregated into apps that help government spooks abduct your neighbors. The effort remains constant, the threats it helps you avoid are getting more dire. Eventually those lines will cross.
In terms of self-hosting vs remote-hosting regular software… I mean, a lot of regular software is fundamentally about communication (social media and office suites) so that goes on the server. LLMs don’t benefit much from being remotely hosted, other than that it makes the flops run in somebody else’s machine. But that will be hard to fund in a universe where the upper-limit to what can be charged is defined by how much additional cost “good AI hardware” adds to the price of your phone. We’ll see I guess.
While this doesn't remove the upfront cost (which could be fixed with basic financing like Apple does) and physical manipulation (e.g. replacement on failure, etc.) it does lower the barrier.
Finally, there's another path I find interesting and interested to think more of which is sharing hardware locally, from your house to your neighborhood, etc. We've done this in the late 90s - early 00s with Internet and it worked really well. I think it could work great for compute.
I see a lot of opportunities in this direction, actually and most are actually enabled by the latest improvements in AI, so most of the "old" thinking doesn't apply anymore.
By that reasoning you'd also conclude that no one cares about affordable housing or having a good work life balance or having your own lake made out of solid gold.
I really do not care if everyone does it my way. What matters is that the choice exists for those that want more ownership, freedom, and privacy.
That said, most next gen laptops will be shipping with capable inference hardware more than most consumers will ever need. Not using the GPU people have in front of them would be silly, but people are silly.
Think of who uses local AI, it’s either
-power users upset with the limits —-cost of intelligence is going down
-people and corps that want stabiltiy —-a massive self own by the labs, this is justified
-American labs seemingly snitching on their users (wtf?) —-huge incentive for people to use Chinese inference, which is cheaper along with other benefits
Regardless, we’re talking about a subset of a subset of people, 1% of 1%. The big bet is that everyone will become AI power users. I think it’s accurate, forget openAI, fuck Anthropic, let’s look at us, the people. Entire workflows like coding have been buried, there’s no going back to how things were before. Is it reasonable to say that will happen to the other work streams? There’s no indication not, and it’s a total gamble if it’s yes
More broadly, those prompts and the responses have value. There's a whole shadow economy in China of aggregating queries across Claude (etc) and then using that as training data. It's like your entire search history is for sale.
Clould infra has typically been opaque to the provider. You run an MySQL instance on an Amazon VM and Amazon doesn't have easy access to the unencrypted traffic and what you store. Cloud LLMs don't operate that way.
AI hardware is improving at a rapid pace. By 2030 I believe a significant amount of LLM usage will be local for these and other reasons.
But they're ignoring that a local LLM can still be very useful for any kind of repetitive task, things like categorization or generating embeddings, translation or summarization tasks. I'm self-hosting Karakeep and use a local LLM for the AI tasks like embedding.
And once you've got that local LLM running, you can start thinking about integrations with your own custom built apps.
Anecdotally, I believe the open weights local models are constantly catching up with commercial ones, but I've honestly lost track. This is a dynamic space.
Some recent tasks I've thrown it:
* Go through my blood test data (from EHR XML dumps from prior years) and give me some over-time charts for consistently-sampled measurements
* Go through my Good Articles bookmarks, epubify them, and drop one down onto my ereader each morning for my commute, then archive the offline version into cold storage
* Go through my music library, identify my most-listened low-bitrate tracks, and source or assemble a shopping cart to get high res versions
* Analyze local tax/foreclosure auctions for vacant land; grab aerial screenshots, tax data, ownership data, and compile a weekly report of things maybe worth bidding on based on some specific selection criteria (fun facts: if you're willing to accept some headaches, 5+ acres can be yours for under $5k, if you're in the right area)
* Dump my grocery store loyalty data, analyze my purchasing habits, and prepare a weekly instacart for my approval that aims to sync with usage habits. Tweak and learn volume/order adjustments over time.
* Given a KVM that can mount ISOs onto the host, reimage two old laptops of unknown status with Debian and install Minecraft so my nieces and I can have a LAN party
* Organize my music (high confidence MusicBrainz Picard matches), movies/TV shows, ebooks, and video game ROM collections with metadata and good folder organization
* Archive bookmarked fanfiction for my offline library (slowly, and respectfully, and with an actual browser. The browser is anti-bot-countermeasured to the gills, but QFN regularly solves hCaptcha and Cloudflare turnstile challenges. Once I gave it the ability to drag, it even reasonably reliably solves drag-shape captchas. A little while back it made a purchase on eBay which is NOT a bot friendly site.)
Would I personally spend time on doing most of these? Generally not. Could an LLM screw them up so badly I regret it? Also generally not. I get the luxury of a psuedo-assistant that doesn't judge how menial the tasks I send it are, and I save time. With a local LLM, my privacy concerns are far less than with frontier models, and local abliterated models are less (or just not at all) twitchy around things that frontier classifiers balk at.
I'm hopeful that big projects like Muse can help to normalize menial task offloading, and that local LLMs provide the bridge for more technically minded folks to use them even more aggressively and limitlessly (as in compute limits/subscription costs) and cheaply.
I feel like people who consider it as an option thinks of narrow scenarios vs ai is part of our life like internet is. How do you things on mobile for example with your local models?
AI is very competitive, I use various models, I like to be on cutting edge and to be productive, its the opposite from doing something permanent, and not changing, where local models can be good. I like to do things on the go, and the last thing any normal person would want is to have DC at their basement.