Hacker News new | past | comments | ask | show | jobs | submit
mikeshi42 | on: Edge0-145B: Frontier AI Capabilities on Your Own Computer
> Sharing matrices substantially reduces the weights that must be stored and moved while keeping the expert count unchanged. Each group of 64 experts shares two large matrices, while each expert retains a small set of parameters that captures its differences.

145B parameters with only peak 29GB memory usage - really impressive, truly something you can actually run on a decent spec MBP

Aurornis | parent | next
Their edge0-strata engine is just a fork of the strata project, which has been doing this for a long time: https://github.com/Niko1221/Strata

They're not hiding that fact once you get to the repo. I mean it's called edge0-strata and the README says it's a fork of strata.

It's weird to see this marketing web page imply that it was all new and novel work that they did, though.

dust42 | root | parent
And their model is just derived from Qwen3.8-next-flash which you can see when going to their HF page. Not much details to be found about the company apart from the founder who "was named in MIT Technology Review's Innovators Under 35".

Not much innovation here though it seems. More like a marketing stunt to collect VC.

Qwen3.8-next-flash is impressive though, especially with Strata or Basalt as inference engines. On a 5090 laptop with 64GB RAM one can run NVFP4 quants at 120-140t/s decode and 3000-4000t/s prefill.

On Mac it is much slower and the Kapitsa engine probably the best bet. Despite the claim of 29GB RAM for Edge0, in order to be fast you need to have at least a 128GB Mac.

vardump | root | parent
How about Splash on Apple silicon? I've heard it's pretty fast.

https://github.com/incoai/splash

wuschel | parent | next
I am not in the material, so perhaps someone else can comment on the quality of the outcome and technique? After skimming a page I am not really sure that the whole model fits inside 29 GB RAM. They seem to use some interesting techniques.

ModelTokenMemoryUsage [Bytes] = Parameters * (BitsPerParameter/8)

For a model with 145B parameters, this means:

    Precision       GB       GiB
    2-bit         36.25     33.76
    3-bit         54.38     50.64
    4-bit         72.50     67.52
    6-bit        108.75    101.28
    8-bit        145.00    135.04
    16-bit       290.00    270.08
    32-bit       580.00    540.17
You have roughly a factor 8 in memory usage from 32 bit down to 4 bit precision. That is quite some room already, but not new.

It seems that they radically reduce the model information required per token via the sparse expert approach - 145B->10B parameters, probably at lower precision - and then use some other approaches I do not understand at that point to make effective use of loaded information.

Did anyone try the model?

loading story #50483169
daemonologist | root | parent | next
In the "Under the Hood of Edge0" section it mentions that it needs another 40 GB of system memory for experts plus 48 GB of SSD storage for vocabulary (engrams?). The 29 GB headline seems to just be a flat lie (/the LLM writing this article getting confused about what memory usage it's reading).
Nursie | root | parent | next
Oh that’s annoying, from the hype I was hoping my 48GB MBP would be able to run this happily, but if we’re talking about -70GB combined RAM footprint then that’s not going to cut it.

Still very interesting though.

loading story #50479575
Tade0 | root | parent
Most likely a huge part of the magic is keeping the ngram table on disk, as only one row from it is read per token, so there's next to no performance hit.

Qwen 3.8 Flash Next utilizes this, with 40% of its parameters being offloaded to disk, making the effective memory requirements much lower.

My daily driver is this:

https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RC...

It takes things a step further by simply chopping off that half of experts which isn't (or rarely is) activated in coding tasks.

End result is a somewhat competent [0] coding model which runs on a 64GB machine.

[0] it still occasionally does weird stuff like replacing an entire file with just one line from its answer when you ask it a question if you don't explicitly tell it to not change anything or loop itself if the KV Cache is quantised at below Q8_0.

origini | parent | next
Don't forget the memory requirements for context.
theroadnotbacon | parent | next
The blog post says:

> For Edge0-145B, day-one support covers macOS, Windows, and Linux, including Apple M-series chips and NVIDIA platforms such as RTX Spark, RTX 4090, and RTX 5090. The release includes one-click deployment tools and hardware guides.

But... there's no guidance on how to run on a mac at all. Weird.

selcuka | root | parent
For some reason the engine that can run the 120B/145B models (edge0-strata) only seems to support CUDA for now:

https://github.com/Edge0-AI/Edge0/tree/main#platforms

loading story #50482877
| parent
{"deleted":true,"id":50478391,"parent":50477029,"time":1794544541,"type":"comment"}