Hacker News new | past | comments | ask | show | jobs | submit
nemo1618 | on: Progress on provable inference for large language models
For open-weight models, provable inference is trivial: just redo the computation! This is much faster, cheaper, and doesn't require any fancy cryptography, just deterministic inference (which, if you read carefully, the ZKP approach needs too). See https://lukechampine.com/repligraphs

Of course, it's unlikely that we'll get open-weight models from OpenAI or Anthropic any time soon. That means we have to choose between a) fancy ZKPs, b) signed responses, or c) comparing results from multiple inference providers.

Whatever method ends up being used, it's clear to me that some form of provable inference is inevitable. People haven't caught on to its more significant applications, but they will. (This post touches upon one of them: model-driven audits.) Think about it: we can prove that a particular mind authored a particular artifact. That's never been possible before! You can't prove that Melville wrote Moby Dick, for example; but you can prove that Claude wrote a poem, or that Gemini approved a pull request, or that GPT got a particular score on some eval. This is a more powerful primitive than a cryptographic signature.

loglog | parent | next
> Of course, it's unlikely that we'll get open-weight models from OpenAI or Anthropic any time soon

This could be worked around by signing the output now and releasing the model after an embargo period, paired with contract penalties if the output of the released model does not match or the cryptographic scheme is compromised in the meantime. However, such guarantees would be worth little, since they can be just moved to a bad bank that defaults on them.

IanCal | parent | next
Is it cheaper to rerun it?

> The verifier runs on the client's CPU. It needs no GPU and only has access to a commitment to the weights. The client downloads the commitment, ~6.5 MB in all, and then one proof per request. Checking a proof takes 17 seconds on 24 threads (and 21 seconds on a single thread), including the offline phase's checks.

As a client that sounds like it might be much cheaper to verify than reprocessing.

PeterisP | root | parent | next
As a client, you're still paying for the compute even if it happens on the service provider's side. Running the same computation twice (perhaps by another provider) is 2x the cost; but if you ask the provider to make this proof, that increases the cost 70-95 times according to the article.

With that level of overhead, all kinds of otherwise inefficient approaches of getting compute start to seem comparably cheap. This is progress, but it still needs like a hundredfold improvement in order to become practical; I don't see a usecase where the benefits of this proof would be worth 70x or even 7x increase in cost.

nemo1618 | root | parent
Since it's CPU vs GPU, it's not apples-to-apples. But in terms of wall-clock time, one inference pass is surely faster than 17 seconds. Plus there's the proof generation step, which the post describes as ~100x slower than generating the response itself.
IanCal | root | parent
21s on a single core for 1024 tokens, that’s ~50t/s to verify and a 6.5 meg download. I’m pretty sure you won’t hit that for a 400B model at 4 bits, but then maybe you would I don’t track that performance so much. You’d need an enormous amount of ram to handle it though.

Yes, the generation is slower but I think here the verification itself is more efficient than regenerating the output.

nemo1618 | root | parent
The bigger factor is that (IIUC) ZKP-based verification is constant-time in the size of the output, so it definitely beats regen-based verification (as long as the constant is reasonable). And the ability to verify on consumer hardware instead of a beefy GPU is important too.

I think there is a place for both. Deterministic inference has other benefits, like "compressing" an output into input+model, as well as more esoteric applications like solving coordination problems. But if we get fast ZKPs for real frontier models, I'll be overjoyed!

baq | parent | next
Good luck rerunning the computation done on 8xB100 unless you aren't a cloud provider yourself