Hacker News new | past | comments | ask | show | jobs | submit
wuschel | on: Edge0-145B: Frontier AI Capabilities on Your Own Computer
I am not in the material, so perhaps someone else can comment on the quality of the outcome and technique? After skimming a page I am not really sure that the whole model fits inside 29 GB RAM. They seem to use some interesting techniques.

ModelTokenMemoryUsage [Bytes] = Parameters * (BitsPerParameter/8)

For a model with 145B parameters, this means:

    Precision       GB       GiB
    2-bit         36.25     33.76
    3-bit         54.38     50.64
    4-bit         72.50     67.52
    6-bit        108.75    101.28
    8-bit        145.00    135.04
    16-bit       290.00    270.08
    32-bit       580.00    540.17
You have roughly a factor 8 in memory usage from 32 bit down to 4 bit precision. That is quite some room already, but not new.

It seems that they radically reduce the model information required per token via the sparse expert approach - 145B->10B parameters, probably at lower precision - and then use some other approaches I do not understand at that point to make effective use of loaded information.

Did anyone try the model?

daemonologist | parent | next
In the "Under the Hood of Edge0" section it mentions that it needs another 40 GB of system memory for experts plus 48 GB of SSD storage for vocabulary (engrams?). The 29 GB headline seems to just be a flat lie (/the LLM writing this article getting confused about what memory usage it's reading).
loading story #50479342
loading story #50478992
Tade0 | parent
Most likely a huge part of the magic is keeping the ngram table on disk, as only one row from it is read per token, so there's next to no performance hit.

Qwen 3.8 Flash Next utilizes this, with 40% of its parameters being offloaded to disk, making the effective memory requirements much lower.

My daily driver is this:

https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RC...

It takes things a step further by simply chopping off that half of experts which isn't (or rarely is) activated in coding tasks.

End result is a somewhat competent [0] coding model which runs on a 64GB machine.

[0] it still occasionally does weird stuff like replacing an entire file with just one line from its answer when you ask it a question if you don't explicitly tell it to not change anything or loop itself if the KV Cache is quantised at below Q8_0.