Hacker News new | past | comments | ask | show | jobs | submit
adgjlsfhk1 | on: AMD acquires Taalas to boost inference performance by etching models in silicon
I don't think this works out from a cost/silicon perspective. Small models already run pretty well in software (since the weights fit in cache) and big models require silicon area proportional to the size of weights. On a mobile device putting a chip like this is competing directly in BOM and power against a whole lot more l3 cache, and the l3 cache makes everything faster
trebligdivad | parent | next
What, even if it means you can run models without relying on the currently backlogged DRAM production?
adgjlsfhk1 | root | parent
The size of model we're talking about running doesn't need much if any dram.
teaearlgraycold | parent | next
My question is what changes about LLM use cases when you’re getting 1000 tok/s? Models in silicon might dramatically change how we think about them.
QuiDortDine | root | parent | next
Did you use chatjimmy? It's somewhat terrifying to use when you think of the potential results with a better model.

Ok, real life example: I now spend most of my time, as a developer, waiting for the agent to do its thing (after careful prompting, I'm also thinking about work stuff, don't worry I'm not useless). What if it gave back the same excellent results, but instantaneously? Why, then, I certainly would become the bottleneck. So, quite possibly, my last work task would be to plug this agent directly into the ticket system where the domain experts input their feature requests. Maybe we still need 1 developer out of 100, to coordinate releases and all that (ok, say 1 out of 10).

But that's not taking things far enough: why do we need these domain experts at all? Our pitch is clear, and all software-enabled, though it took years to develop. We can just have the clients express their concerns to the AI, directly or indirectly. Have multiple lighting-fast agents with different roles (refactoring agent, new features agent, debugger agent, domain expert agent, etc.). So we fire everyone, maybe keep 1 product owner / devops to keep the trolls out. The cost is still probably 100 times less than it used to be (beyond the initial cost of acquisition of the magic machine or whatever).

But one of these clients, surely, will realize that these 10 years of manual and slowly-automated development can now be emulated in very, very little time. Why not just, say, take screenshots of the entire app and feed them into the magic machine? Why, this way, they could have the service for a tenth of the yearly cost, forever!

And then the economy implodes.

I'm not saying it's THE most likely version of things, I'm saying that at a certain level, quantity (or rather, speed) is a quality all its own. And this new quality might change the world. Let's hope it's for the better!

loading story #49205783
loading story #49204762
loading story #49205279
loading story #49204623
HDBaseT | root | parent | next
In the case on on-device/self-hosted LLMs. You ask your agent to implement xyz feature 10 times and use a model to compare the outputs and combine the best results.

Raw intelligence becomes slightly less important when you can iterate and improve automatically. You can still claim it was "one shot" even when 30 different implementations were made then combined.

in_a_society | root | parent | next
The best way I can explain it is that it's the same feeling when I upgraded from 56k dialup to cable broadband.
LarsDu88 | root | parent | next
Massive economic simulations with thousands if not millions of agents to front run the global economy and stock market.

Fully interactive realtime NPCs in videogames at scale.

Recommender systems that simulate individual consumers.

Crazy shit

RussianCow | root | parent
That likely isn't as relevant for on-device iPhone usage as it is for Real Work™. I won't notice the difference between 50tps and 1000tps when asking Siri a question.
spijdar | root | parent | next
I don't know. As others have said, the Taalas chip wasn't small, or particularly low power, so it's hard to "imagine" what that tech in an cell phone chip might look like.

But if the basic premise of "good enough LLM at insane throughput" holds, I think it could qualitatively change local uses of LLMs. At a certain speed point, you're able to move from request -> response to a cascade of tool calling and "subagents", which could allow a small model to be much more useful, if provided with a lot of local data and tool calls.

That said, this is assuming you could stuff a "good enough" model into a phone with Taalas-like technology. The Taalas tech demo was an 8B parameter model and required hundreds of watts (IIRC) to run. The efficiency was good given the speed (as I understand), but it's not clear at all that the approach scales small enough to be a sensible coprocessor on an iPhone or whatever.

loading story #49203964
loading story #49203973
loading story #49204141
dboreham | parent | next
Works great from a press release perspective though.
bastawhiz | parent
The weights might fit in cache, if you're using a small model. If you wanted to have a 20B+ parameter model, that's just going in RAM. You could put more RAM in the device and pay the perf cost or have a dedicated chip. Most devices already have a dedicated chip, this just changes which silicon you're spending the money on.
wmf | root | parent
That math doesn't really work.

8B model (FP4) = 4 GB DRAM = 32 Gb DRAM = 80 mm2

8B model (Taalas) = 4 GB ROM = ~800 mm2