Hacker News new | past | comments | ask | show | jobs | submit
msdz | on: Qwen3.8-Max: A New Bar for Coding and Cowork
Not who you responded to, but I’ll give it a try.

Imagine our LLM as this one-dimensional matrix (instead of unfathomably many, many more dimensions):

0.728201017

-1.9293839303

2.93020202929

5.82290101728

-0.2844920917

…

And then, all of these values are multiplied by some scaling factor, essentially vectors pointing in some direction (in our case, only on a one-dimensional line instead of in some higher-dimensional space).

Now imagine it’s possible to “remove” the portion of the weights’ directions that’s responsible for some undesired property like history knowledge, or another programming language:

0.428201017

-1.6293839303

2.63020202929

5.52290101728

-0.5844920917

…

(For demonstration and faster-to-type-out purposes, I’ve changed each vector by deducting a constant amount of 0.3, which doesn’t make much sense. In reality, it’d modify probably all digits in some direction.)

Now we can see: the number of rows has stayed the same, but the value has changed. This means the required storage (or computational) space remains the same.

Please do correct me if this simplified too much or I had a mistake in my understanding.

gcr | parent | next
This is oversimplified. The proposed question is whether a model with fewer parameters could achieve performance on one language similar to that of a larger model that’s been trained more broadly, which isn’t straightforward to do.
loading story #49156015
mattnewton | parent
The exception being the token embeddings and lm head (which scale with the number of tokens the model knows and presumably you need a smaller number in the tokenizer for only English and python). But those are a pretty small % of the total model weights on most LLM sizes