This is expected if you're actively steering the agent, but what about running unattended tasks - perhaps multiple such tasks in parallel, such that you can stay engaged by "attending" to each of them in turn? Then you're not restricted to such a requirement, and can benefit from much slower (individually, not wrt. total throughput) inference streams.
> I currently have 192GB and 24GB VRAM
I'm not familiar with beellama, but this seems like it could be the perfect setup for Strata which people are saying good things about, including for running DS4.1F (at least quantized). The whole point of it is to effectively leverage heterogenous RAM/VRAM setups. Of course llama.cpp itself is doing a good job of trying to catch up, but they've yet to support efficient SSD streaming (mmap doesn't quite cut it) in the main branch.
How so? This is not obvious to me; AIUI Strata does support streaming weights for larger models via SSD. Of course the speed would drop quite a bit once you rely on SSD, but it ought to be workable. Since you're on a HEDT/server platform, you can also benefit by adding more SSD hardware to be accessed in parallel via the abundant PCIe lanes.