Hacker News new | past | comments | ask | show | jobs | submit
I don't know at all. But I've been playing with oMlx and Splash. Using Pi there are compression extensions like the billion-context. I suspect your right.

But all seem to have massive issue boards. What I did see was oMlx release clustering so you can string 3 macs and have 3 times the memory or use one for prefilling accelerating.

Which I thought was neat and what I'll likely poke at instead of getting more ran.

perhaps i'm not pilled enough for this conversation, but aren't you conflating two disparate concepts — model-weights and context-window?
Yah I think I am conflating it. I was thinking using the ram more effectively to begin to squeeze more, one of those is context size. Less VRAM the less context you can handle, so there always a lot of tuning to find that sweet spot.

So my mind on that.