> We found that the model's modulation weights (~40% of the total parameters) could be pruned and replaced with a functionally equivalent lookup table, dramatically shrinking the memory footprint with no loss in output quality.
Is this a common approach to reducing weights with "no loss in output quality", assuming this is true? Seems almost too simple to work. If this is doable, would this be applicable to LLMs as well?
Neat with native frame-to-frame generation, but wonder how easy it is to "link" together clips at the intersection, typically the models kind of lose the "momentum" across these stiches, being able to merge things with frame-to-frame between clips might help with this it feels like.
It is a well-known trick, given that the timestep is between 0 to 1, you can slicing them at any resolution (1000, or 10000, give or take), and then keep a look-up table for modulation scale / bias etc for each. It is quite different from quantization and it is indeed lossless.
It is also only applicable to diffusion models as only these operates at per-timestep.
loading story #49159276
"Modulation weights" here refers to weights used to adjust layer normalization depending on the task (adaLN). General-purpose LLMs generally don't have those in the first place.
loading story #49158938
Also begs the question whether this is applicable for high-throughput applications on FPGAs, which are to my novice mind basically LUTs, right?
I remember a paper which was posted on HN a few weeks ago where somebody implemented KAN networks in FPGAs, since those can readily be approximated as LUTs.
Whoah, could this mean we can treat layers like a jpg, where we come up with a formula that estimates the weight values of a layer instead of storing all of the weights?
loading story #49157871