This is a bit different though. It’s using transformer architecture to emulate almost at the circuit level. It’s probably even more inefficient than something that was trained to emulate a cpu, but with the advantage of actually being an exact emulation rather than learned approximation.

It’s a bit like when a human brain remembers facts, if you sat down and computed the hash values of the thing you were trying to remember, and traced out a DB lookup in your mind, it’d be a lot slower than just “oh yeah I remember that”.

> A while back I saw some people saying we would just have LMMs hallucinate an entire operating system every single frame. I guess technically it's now proven to be technically possible, but it would be thousands of times slower while requiring thousands of times more compute (a multi-million x difference from an ordinary OS!).

Gosh. That would be a very silly use of an LLM.

Bad use of any AI, even image/video models which are certainly less wrong than this. Stuff like DLSS show we're getting stuff that could generate a game's graphics frame-by-frame, but even though it's possible to have an entire game environment (not just graphics, everything) done by an AI, such things demo better than they play due to temporal coherency.

That said, having an AI make the code for each thing as you need it, might work somewhat OK eventually. Wouldn't trust it today (security, data integrity, and a tendency to cheat on tests), but eventually it could happen.

A middle ground would be an LLM taking structured inputs ("mouse clicked at X,Y") and making "draw" calls at whatever level: pixels, SVG, HTML/CSS. While the LLM's output tokens could be mapped to just these calls, it could also just do a bunch of tool calls that write to these formats as well -- it's a matter of efficiency.

We have Jev to prove LLMs that output constrained decisions are useful. Why not make LLMs just output UI natively?

At a fast enough inference speed, you could just connect the network to it.

So an LLM would then just take the network, user input, and output a UI, with tool calls to deal with memory and other resources. In theory, you don't need an OS any more.

They've already forced Electron on us, I'm sure they'll make LLM-approach just good enough too.

This is exactly like if someone implemented a CPU using nothing but CSS animations. Maybe a fun way to teach about turing completeness, but pretty meaningless otherwise.

> show the massive compute requirement between LLMs and ordinary programs completely agree, but it is at least raising the point that 150-300x is only a bit wider than things like framework/language choice... which is quite remarkable considering how much more open-ended running everything through an LLM is.

> hallucinate an entire operating system every single frame 1. I don't know that "hallucinate" is the right word here... any more than Windows 11 itself is a hallucination, for example 2. I don't personally think that this system is a million (thousands x thousands, from your attestation) times slower, but given Moore's Law, even that isn't the worst thing ever. this is the low water mark.

Let's say you are developing a product, choose 1:

1) Have a single person vibe together a computationally inefficient solution in 1 month that runs transformers on some backend.

2) Hire a team to work for 2 years to design, perfect, optimize, plan, revise, and architect a solution.

The risk/reward for 1) is extremely better than 2). Given the turnover for entertainment in this day and age it doesn't make sense to do all this complex stuff.

> have LMMs hallucinate an entire operating system (...) technically it's now proven to be technically possible

You're comparing apples to oranges there!

To hallucinate is arbitrary, you can use a 2B image param model, it will hallucinate an OS no problem* running real time!

What this post is about is completely different, which is actually emulating the complete system in an LLM - that is hundreds (or more) times slower.

*absulutely crappy quality tho, unusable OS in practice

That's the case right now--but what might a customized model tailored only for a narrow set of OS functions behave like? I suspect its achievable with a lot less compute.

Now is that a desirable product? Would it succeed on the merits in the marketplace? Maybe it would...

These systems in weights are proof of concepts. Surely there is room for optimization in there somewhere.

Isn't that the same with e.g. human brain? Computer program can compute billions of equations, whilst human will struggle with just one.

It would be a natural extension of the current work in software. Every layer of abstraction/web/scripting language adds seemingly 10x in both ram and processing time, undoing the advancement of 1000x faster processors and ram increases.

But at some point we’re hitting a wall here. If we want our software to undo 100,000x more processing powers, we need that llm software emulation layer.