Hacker News new | past | comments | ask | show | jobs | submit
bensyverson | on: AirLLM 70B inference with single 4GB GPU
And is this using conventional model loading (all in VRAM), or are you streaming it in some way?