Chinese labs seem to think that fable/mythos is a 10T+ model, and target that amount of parameters with next iteration of qwen/kimi/ds/glm. Proves nothing tho.
With common use of n-gram tables, comparing "params" count has gotten harder. Those are exceptionally sparse and light on compute, way beyond even the sparsest MoE experts. You could host them on a slow spinning disk and still have a viable local inference platform.
So that's why I can't buy a replacement 16TB HDD for less than a 1000$ when three years ago I bought them for 229$ a pop.
I wondered why do they even need slow HDDs, like, do they store training data on them? User chats? Do they assemble crazy 12-way stripes for fast speed? Do they store what, failed attempts?
I think that’s just because hoovering up the entire worlds worth of data and saving it to a RAID multiplied by every frontier lab is a lot of fucking hard drives.