Existence proof: everything Claude Code does, including talking to LLMs on GPUs via the Anthropic API, and shelling out to other tools installed on your system, could in principle be compiled into one big self-contained binary (or Turing Machine if you prefer) that is able to take natural language in and produce compiled binaries out.
If you wanted to go a bit harder you could train Claude to output machine code directly, but that's not necessary. Whether the machine code comes straight out of an LLM or out of a compiler that is fed source code that comes from an LLM is a mere implementation detail, there's not any fundamental difference.
There are also times when you want the final product to be able to target different architectures and platforms. Generating into a higher level language allows for re-targeting the code by only changing a few compiler & linker flags or runtime/environment config. I would much rather that than hoping the LLM is feature- (and bug-)compatible between generation runs.