Hacker News new | past | comments | ask | show | jobs | submit
xbar | on: Flash-MoE: Running a 397B Parameter Model on a Laptop
0.1 GB per full-attention layer and "The model has 60 transformer layers: 45 GatedDeltaNet (linear attention) + 15 standard full attention." So, 1.5 GB.